From 10bee38e3c2678bb74d7ac4e5189f50ede9493bc Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Wed, 26 Aug 2026 17:31:23 +0300 Subject: [PATCH 01/16] feat(sandbox): give each candidate its own agent-CLI state Every candidate container mounts the shared helix-auth- login volume at /home/node read-write. That is deliberate and unchanged: it is what lets a CLI refresh its token and take its cross-process refresh lock. But the CLIs also write their working state there, so candidate N started life reading candidate N-1's transcripts, session databases, memories and to-do lists. For an optimizer whose candidates are meant to be independent samples that contaminates the experiment, and some of that state (opencode's opencode.db) carries token columns as well. Mount a second, per-candidate directory at /helix-state -- outside /home/node, so the auth mount is untouched -- and point each backend's state at it. It lives in the sandbox's existing temporary tree, so it is created and removed with the candidate. Three backends have a knob that moves state without moving the credential; all three were verified against the real CLI in a container with a synthetic credential: codex -c sqlite_home=... state_5.sqlite, logs_2.sqlite opencode OPENCODE_DB=... opencode.db (+ -wal/-shm) cursor CURSOR_CONFIG_DIR= the whole ~/.cursor tree Choosing the knob is the whole problem: the obvious environment variable usually relocates the credential too and silently breaks login. XDG_DATA_HOME does this to opencode (auth list then reports 0 credentials) and XDG_CONFIG_HOME does it to cursor (status then reports Not logged in). Both are recorded in REJECTED_AGENT_STATE_KNOBS so they are not re-tried, and pinned by container tests. Unsandboxed opencode runs used XDG_DATA_HOME for per-candidate SQLite isolation, which hid any existing opencode login. Switch them to OPENCODE_DB; the on-disk layout is unchanged. claude is deliberately not isolated. CLAUDE_CONFIG_DIR moves the credential with the transcripts, and masking its state subdirectories was evaluated and rejected -- the mask list is already stale against the shipped CLI, it writes to the shared volume, and it silently breaks transcript preservation. docs/agent-state-isolation.md has the detail. Residue that still crosses candidates, including codex's session rollouts, is named in UNRELOCATED_AGENT_STATE rather than left implicit. Co-Authored-By: Claude Opus 5 --- README.md | 51 +++ docs/agent-state-isolation.md | 92 ++++++ pyproject.toml | 1 + src/helix/agent_state.py | 181 +++++++++++ src/helix/mutator.py | 66 ++-- src/helix/sandbox.py | 52 +++ tests/integration/__init__.py | 0 tests/integration/conftest.py | 130 ++++++++ .../integration/test_agent_state_isolation.py | 296 ++++++++++++++++++ tests/unit/test_agent_state.py | 272 ++++++++++++++++ tests/unit/test_mutator.py | 45 +-- tests/unit/test_sandbox.py | 11 +- 12 files changed, 1158 insertions(+), 39 deletions(-) create mode 100644 docs/agent-state-isolation.md create mode 100644 src/helix/agent_state.py create mode 100644 tests/integration/__init__.py create mode 100644 tests/integration/conftest.py create mode 100644 tests/integration/test_agent_state_isolation.py create mode 100644 tests/unit/test_agent_state.py diff --git a/README.md b/README.md index 8f0af146..f3bfe4f0 100644 --- a/README.md +++ b/README.md @@ -637,6 +637,57 @@ session. The volume names are `helix-auth-claude`, `helix-auth-codex`, `helix-auth-cursor`, `helix-auth-gemini`, and `helix-auth-opencode`. + +#### Per-candidate agent state + +The auth volume is shared by every candidate container on purpose: it is what +lets a CLI refresh its token and take its cross-process refresh lock. But the +CLIs also write their *working state* under `/home/node` — transcripts, session +databases, memories, to-do lists — so without further work candidate N would +start by reading candidate N-1's leftovers. For an optimizer whose candidates +are meant to be independent samples, that contaminates the experiment; some of +that state (opencode's `opencode.db`) also carries token columns. + +HELIX therefore mounts a second, per-candidate directory at `/helix-state` — +deliberately outside `/home/node`, so the shared auth mount is unchanged — and +points each backend's state at it. The directory lives in the same temporary +tree as the sandbox workspace copy, so it is created and deleted with the +candidate. `helix.agent_state` holds the knobs: + +| Backend | Knob | Moves | Credential | +| --- | --- | --- | --- | +| `codex` | `-c sqlite_home=…` | `state_5.sqlite`, `logs_2.sqlite` (+`-wal`/`-shm`) | `.codex/auth.json` stays shared | +| `opencode` | `OPENCODE_DB=…` | `opencode.db` (+`-wal`/`-shm`) | `auth.json` and the lock dir stay shared | +| `cursor` | `CURSOR_CONFIG_DIR=…` | the whole `~/.cursor` tree | `~/.config/cursor/auth.json` stays shared | +| `claude` | none | — | see below | + +Picking the knob matters: the obvious environment variable is usually the wrong +one because it relocates the credential too, which silently makes an existing +login invisible. `XDG_DATA_HOME` does this to opencode (`opencode auth list` +then reports `0 credentials`) and `XDG_CONFIG_HOME` does it to cursor +(`cursor-agent status` then reports `Not logged in`). HELIX never sets either; +if you route `XDG_CONFIG_HOME` through `passthrough_env` or `[env]`, the cursor +backend logs a warning because it will break that backend's login. +`helix.agent_state.REJECTED_AGENT_STATE_KNOBS` records these so they are not +re-tried, and `tests/integration/test_agent_state_isolation.py` pins the +behaviour against the real CLIs. + +**What still crosses candidates.** Relocation is partial, and the residue is +listed per backend in `helix.agent_state.UNRELOCATED_AGENT_STATE`. The +significant cases: + +- **codex** keeps writing its session rollout transcript to + `.codex/sessions//rollout-*.jsonl`, plus `shell_snapshots/` and + `memories/`, in the shared volume. `sqlite_home` does not cover these and the + CLI exposes no separate knob for them; only `CODEX_HOME` moves them, and that + moves `auth.json` with them. +- **opencode** keeps `log/*.log` and `storage/session_diff/ses_*.json`. +- **claude** is not relocated at all. `CLAUDE_CONFIG_DIR` is all-or-nothing — + it moves `.credentials.json` together with the transcripts — and the + alternative of mounting empty per-candidate volumes over + `.claude/projects`, `.claude/sessions`, `.claude/telemetry` and + `.claude/backups` was evaluated and rejected; see + `docs/agent-state-isolation.md`. This avoids copying host credential stores into Docker. On macOS, Claude/Cursor browser-login tokens may live in Keychain; on Linux they may live in Secret Service/libsecret, GNOME Keyring, KWallet, or another desktop keyring. diff --git a/docs/agent-state-isolation.md b/docs/agent-state-isolation.md new file mode 100644 index 00000000..c413eca7 --- /dev/null +++ b/docs/agent-state-isolation.md @@ -0,0 +1,92 @@ +# Per-candidate agent state, and why claude is not isolated + +HELIX mounts one login volume per backend (`helix-auth-`) at +`/home/node`, read-write, in every candidate container. That mount is shared on +purpose and is not negotiable: it is what lets each CLI refresh its token and +take its cross-process refresh lock across concurrent candidates. + +The problem this document is about is everything *else* the CLIs write into +that volume. `helix.agent_state` relocates what it safely can to a +per-candidate directory mounted at `/helix-state`; the README's "Per-candidate +agent state" section covers the three backends that worked. This note records +the reasoning for the one that did not, so it does not get re-litigated from +scratch. + +All observations below are from the images HELIX ships +(`ghcr.io/ke7/helix-evo-runner-*:latest`), against synthetic credentials in +throwaway volumes. + +## Why `CLAUDE_CONFIG_DIR` is not usable + +It is all-or-nothing. It relocates `.credentials.json` together with the +transcripts, and it additionally pulls `.claude.json` into whatever it points +at. Pointing it at a per-candidate directory would give each candidate a clean +state tree and no credential, which defeats the entire purpose of the shared +login volume. There is no second knob: of the `CLAUDE_*` variables the CLI +understands, none relocates state alone. + +## Why masking was evaluated and rejected + +The alternative is *masking*: leave the config directory shared and mount empty +per-candidate volumes over its state subdirectories. This was tried against +Claude Code 2.1.138. It does relocate state — `projects/`, `sessions/`, +`telemetry/`, `backups/` and the contents of `.claude.json` all landed in the +per-candidate directory. It was still rejected, for four reasons. + +**1. The mask list has to be maintained against the CLI, and is already +stale.** The subdirectories a reasonable person would name — `projects/`, +`sessions/`, `todos/` — are not the ones this version writes. There is no +`todos/` at all, and there are two that the obvious list misses: `telemetry/` +and `backups/`, both of which carry per-session identifiers. A mask list that +is already wrong for the currently shipped CLI is the clearest possible +evidence that it will drift again, and each drift is silent: a newly added +state directory simply starts leaking between candidates with nothing to +signal it. + +**2. `.claude.json` is a file outside the config directory.** It lives at +`$HOME/.claude.json`, not under `.claude/`, so masking it needs a *file*-level +bind mount rather than a directory one. The CLI also rewrites it through a +backup-and-replace cycle — a `.claude/backups/.claude.json.backup.` +appears on every run. File bind mounts do not survive an atomic +rename-into-place, so this is a mechanism that works until the day the CLI +changes how it saves that file, and then fails in a way that is hard to +attribute. + +**3. Masking writes to the shared volume, which the isolation work is not +allowed to do.** A bind mount needs its mountpoint to exist, and Docker creates +it inside the volume. Masking the four directories plus `.claude.json` added +five new entries to the shared login volume, including turning `.claude.json` +into a 0-byte file there. The knob-based approach used for codex, cursor and +opencode leaves the shared volume byte-for-byte unchanged; masking cannot. + +**4. It silently breaks transcript preservation.** +`helix.sandbox._copy_claude_transcript_from_auth_volume` recovers the session +transcript by starting a *separate* container that mounts only the auth volume +read-only and copies from `sandbox.claude_transcript_root`. That container does +not carry the agent container's masks, so once `projects/` is masked the +transcript it is looking for is no longer in the volume. The helper's +`[ -f "$src" ] || exit 0` guard means this fails silently: +`preserve_backend_transcripts` would keep reporting success while saving +nothing. + +## The verification gap + +Independently of the above, requirement (c) of this work — *demonstrate the CLI +still reports itself authenticated* — cannot be met for claude without a real +grant. Claude Code validates the credential's shape before reporting status, so +a synthetic credential yields `Not logged in · Please run /login`. That is not +caused by masking (an unmasked container with the same synthetic credential +reports exactly the same thing), but it does mean the only way to prove a +claude change is safe is to run it against a live login. Proving isolation by +risking the credential it is supposed to protect is a bad trade. + +## Conclusion + +Claude is left exactly as it is. Its cross-candidate residue is recorded in +`helix.agent_state.UNRELOCATED_AGENT_STATE` under the `claude` key so that it +is discoverable rather than forgotten. Three of four backends are isolated; +this one is documented instead. + +Anyone revisiting this should start by re-running the footprint check against +the current CLI, because the specific directories named above are version +facts, not stable API. diff --git a/pyproject.toml b/pyproject.toml index 1110dc24..e1be41b9 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -83,4 +83,5 @@ timeout = 60 timeout_method = "thread" markers = [ "diff_harness: differential-testing harness (phases 2-4)", + "docker_integration: runs real containers; needs a Docker daemon and the backend images", ] diff --git a/src/helix/agent_state.py b/src/helix/agent_state.py new file mode 100644 index 00000000..64d12747 --- /dev/null +++ b/src/helix/agent_state.py @@ -0,0 +1,181 @@ +"""Per-candidate relocation of agent-CLI state, away from the shared auth volume. + +Why this module exists +---------------------- +HELIX mounts one login volume per backend (``helix-auth-``) at +``/home/node`` read-write in *every* candidate container. That mount is what +keeps the CLIs' token refresh and their cross-process refresh locks working, so +it is deliberately shared and must stay exactly as it is. + +The problem is what else rides along in that volume. Each CLI also writes its +*agent state* under ``$HOME`` -- transcripts, session databases, memories, +to-do lists. Because the volume is shared, candidate N starts life reading +candidate N-1's leftovers. For an evolutionary optimizer whose candidates are +meant to be independent samples, that is contamination of the experiment. A +second, smaller consequence is that some of this state doubles as a credential +store (opencode's ``opencode.db`` carries ``access_token`` / ``refresh_token`` +columns), so it should not be lying around in a shared location either. + +What this module does +--------------------- +For the backends that expose a knob separating *state* from *credential*, it +returns the environment variables and CLI arguments that point state at a +per-candidate directory. The credential file is never named, never moved and +never copied: it keeps living in the shared volume exactly where the CLI put +it. See :data:`UNRELOCATED_AGENT_STATE` for what each backend leaves behind. + +Choosing a knob is not obvious, and the wrong choice silently breaks login. +The knobs below were each verified against the real CLI in a container with a +synthetic credential; the rejected alternatives are recorded in +:data:`REJECTED_AGENT_STATE_KNOBS` so nobody re-tries them. +""" + +from __future__ import annotations + +import json + + +AGENT_STATE_CONTAINER_ROOT = "/helix-state" +"""Container path where the per-candidate state directory is mounted. + +Deliberately *outside* ``/home/node``. Mounting anywhere under ``/home/node`` +would place a mountpoint inside the shared auth volume, which creates a new +entry there -- the one thing the shared mount is not allowed to acquire. +""" + + +STATE_RELOCATING_BACKENDS: frozenset[str] = frozenset( + {"codex", "cursor", "opencode"} +) +"""Backends with a knob that moves state without moving the credential. + +``claude`` and ``gemini`` are absent on purpose; see +:data:`UNRELOCATED_AGENT_STATE`. +""" + + +UNRELOCATED_AGENT_STATE: dict[str, tuple[str, ...]] = { + # Paths are relative to ``$HOME`` (the shared auth volume mount point) and + # still carry cross-candidate state after relocation. Named here so the + # residue is discoverable rather than forgotten. + "codex": ( + ".codex/sessions////rollout-*.jsonl", # full transcript + ".codex/shell_snapshots/*.sh", + ".codex/memories/", + ".codex/config.toml", + ".codex/installation_id", + ), + "cursor": (), + "opencode": ( + ".local/share/opencode/log/*.log", + ".local/share/opencode/storage/session_diff/ses_*.json", + ".local/share/opencode/storage/migration", + ".config/opencode/.gitignore", + ), + "claude": ( + ".claude/projects//*.jsonl", # full transcript + ".claude/projects//memory/", + ".claude/sessions/", + ".claude/telemetry/", + ".claude/backups/", + ".claude.json", + ), + "gemini": (".gemini/", ".config/google-gemini/"), +} + + +REJECTED_AGENT_STATE_KNOBS: dict[str, str] = { + # Each of these looks like the obvious knob and each one breaks login. + "opencode:XDG_DATA_HOME": ( + "moves opencode.db AND auth.json together; with it set, " + "`opencode auth list` reports 0 credentials" + ), + "cursor:XDG_CONFIG_HOME": ( + "moves cli-config.json AND auth.json together; with it set, " + "`cursor-agent status` reports 'Not logged in'" + ), + "cursor:CURSOR_DATA_DIR": "accepted but relocates nothing; cli-config.json stays in $HOME", + "codex:CODEX_HOME": "moves the state databases AND auth.json together", + "claude:CLAUDE_CONFIG_DIR": ( + "moves the transcripts AND .credentials.json together, and pulls " + ".claude.json in as well" + ), +} + + +def _backend_state_dir(backend: str, state_root: str) -> str: + """Return the per-backend subdirectory of the per-candidate state root.""" + return f"{state_root.rstrip('/')}/{backend}" + + +def agent_state_subdirs(backend: str) -> tuple[str, ...]: + """Return directories to create under the state root before the container runs. + + The CLIs are not uniformly willing to create a missing parent directory for + a relocated database, so HELIX creates them itself and keeps the behaviour + deterministic across backends. Paths are relative to the state root. + """ + if backend not in STATE_RELOCATING_BACKENDS: + return () + return (backend,) + + +def agent_state_env(backend: str, *, state_root: str) -> dict[str, str]: + """Environment variables that point *backend*'s state at a per-candidate dir. + + Returns an empty mapping for backends without a safe knob, so callers can + apply the result unconditionally. + """ + state_dir = _backend_state_dir(backend, state_root) + if backend == "opencode": + # Verified: relocates opencode.db and its -wal/-shm companions alone. + # auth.json stays at $HOME/.local/share/opencode/auth.json and the + # refresh lock stays at $HOME/.local/state/opencode/locks/. + return {"OPENCODE_DB": f"{state_dir}/opencode.db"} + if backend == "cursor": + # Verified: relocates the whole ~/.cursor state tree (cli-config.json, + # agent-cli-state.json, statsig-cache.json, projects//mcp-auth.json). + # The credential is read from ${XDG_CONFIG_HOME||~/.config}/cursor/auth.json, + # which this knob does not affect. + return {"CURSOR_CONFIG_DIR": state_dir} + return {} + + +def agent_state_cli_args(backend: str, *, state_root: str) -> list[str]: + """CLI arguments that point *backend*'s state at a per-candidate dir. + + Used for backends whose only knob is a config override rather than an + environment variable. + """ + if backend == "codex": + # Verified: relocates state_5.sqlite and logs_2.sqlite (plus their + # -wal/-shm companions). auth.json stays at $HOME/.codex/auth.json. + # + # ``-c key=value`` requires a TOML literal on the right-hand side; + # ``json.dumps`` emits a double-quoted string that is also valid TOML + # basic-string syntax, matching how ``model_reasoning_effort`` is + # passed in ``helix.mutator._build_backend_args``. + state_dir = _backend_state_dir(backend, state_root) + return ["-c", f"sqlite_home={json.dumps(state_dir)}"] + return [] + + +def cursor_credential_hazard(backend: str, env: dict[str, str]) -> str | None: + """Return a warning when *env* would hide cursor's shared credential. + + ``cursor-agent`` resolves its credential to + ``${XDG_CONFIG_HOME||~/.config}/cursor/auth.json``. HELIX never sets + ``XDG_CONFIG_HOME`` itself -- the env scrub in ``helix.executor`` is an + allowlist -- but a user can route it through ``passthrough_env`` or the + ``[env]`` table in ``helix.toml``. If they do, cursor stops seeing the + shared login volume entirely and reports "Not logged in", which is worth a + warning rather than a silent authentication failure mid-run. + """ + if backend != "cursor" or "XDG_CONFIG_HOME" not in env: + return None + return ( + "XDG_CONFIG_HOME is set for the cursor backend. Cursor reads its " + "credential from ${XDG_CONFIG_HOME}/cursor/auth.json, so this hides the " + "shared login volume and cursor will report 'Not logged in'. Remove " + "XDG_CONFIG_HOME from passthrough_env / [env] in helix.toml." + ) diff --git a/src/helix/mutator.py b/src/helix/mutator.py index 4f53c202..e063fd0b 100644 --- a/src/helix/mutator.py +++ b/src/helix/mutator.py @@ -10,6 +10,12 @@ from pathlib import Path from typing import Any, Callable +from helix.agent_state import ( + AGENT_STATE_CONTAINER_ROOT, + agent_state_cli_args, + agent_state_env, + cursor_credential_hazard, +) from helix.backends import BACKEND_AUTH_ENV, backend_display_name from helix.display import UsageStats from helix.population import Candidate, EvalResult @@ -644,9 +650,9 @@ def _ignore_helix_artifacts(worktree_path: Path) -> None: BACKEND_STDERR_ARTIFACT_NAME, ".helix_artifacts/", "helix_batch.json", - # Per-candidate OpenCode SQLite state (XDG_DATA_HOME isolation). + # Per-candidate OpenCode SQLite state (OPENCODE_DB isolation). # Each parallel opencode worker gets a fresh database here; keeps - # the candidate git tree free of opencode's session/session files. + # the candidate git tree free of opencode's session transcripts. ".helix_opencode_state/", ] existing = gitignore.read_text() if gitignore.exists() else "" @@ -734,7 +740,15 @@ def _build_backend_args( worktree_path: str, config: AgentConfig, prompt_artifact_name: str, + agent_state_root: str | None = None, ) -> list[str]: + """Build the backend CLI argv. + + *agent_state_root* is the container path of the per-candidate state + directory when the command runs sandboxed, and ``None`` otherwise. Only + backends whose state knob is a CLI override rather than an environment + variable consume it -- currently just codex. + """ backend = config.backend if backend == "claude": args = [ @@ -776,6 +790,12 @@ def _build_backend_args( args.extend( ["-c", f"model_reasoning_effort={json.dumps(config.effort)}"] ) + if agent_state_root is not None: + # Points codex's state databases at the per-candidate directory. + # ``auth.json`` is not affected and stays in the shared volume. + args.extend( + agent_state_cli_args(backend, state_root=agent_state_root) + ) args.append(_prompt_file_instruction(prompt_artifact_name)) return args @@ -1589,13 +1609,13 @@ def invoke_claude_code( return _MUTATOR_OVERRIDE(worktree_path, prompt, config) backend = config.backend backend_name = backend_display_name(backend) - backend_worktree_path = ( - "/workspace" if sandbox is not None and sandbox.enabled else worktree_path - ) + sandbox_enabled = sandbox is not None and sandbox.enabled + backend_worktree_path = "/workspace" if sandbox_enabled else worktree_path args = _build_backend_args( backend_worktree_path, config, prompt_artifact_name, + agent_state_root=AGENT_STATE_CONTAINER_ROOT if sandbox_enabled else None, ) cmd_str = shlex.join(args) backend_env = _scrub_environment( @@ -1604,30 +1624,36 @@ def invoke_claude_code( _add_backend_auth_env(backend_env, backend) if backend == "gemini": backend_env["GEMINI_CLI_TRUST_WORKSPACE"] = "true" - if backend == "opencode" and (sandbox is None or not sandbox.enabled): + if warning := cursor_credential_hazard(backend, backend_env): + logger.warning("%s", warning) + if backend == "opencode" and not sandbox_enabled: # Per-candidate SQLite isolation for concurrent opencode subprocesses. # - # OpenCode stores its session database at: - # macOS: ~/Library/Application Support/opencode/opencode.db - # Linux: $XDG_DATA_HOME/opencode/opencode.db (default ~/.local/share/opencode/) - # # When multiple proposals run in parallel (num_parallel_proposals > 1), # every worker spawns a fresh `opencode run` subprocess that issues - # `PRAGMA journal_mode = WAL` against this shared database at startup. + # `PRAGMA journal_mode = WAL` against a shared database at startup. # Concurrent WAL-mode requests on the same file produce: # "Failed to run the query 'PRAGMA journal_mode = WAL'" - # (observed in PR #34 E2E re-verify: g1-s1 lost to this error while g1-s2 succeeded). + # (observed in PR #34 E2E re-verify: g1-s1 lost to this error while + # g1-s2 succeeded). # - # Fix: set XDG_DATA_HOME to a per-candidate directory. OpenCode respects - # XDG_DATA_HOME and will create an isolated database at: - # /.helix_opencode_state/opencode/opencode.db - # Each parallel worker gets its own fresh database; no contention. + # The knob is OPENCODE_DB, which relocates opencode.db and its + # -wal/-shm companions and nothing else. XDG_DATA_HOME would also + # work for the locking problem but moves auth.json with the database, + # which makes an existing opencode login invisible -- verified against + # the real CLI, where `opencode auth list` then reports 0 credentials. # - # The sandbox branch is excluded: container isolation already provides - # per-candidate filesystem separation, so XDG_DATA_HOME would be redundant. + # The resulting layout is unchanged from the previous XDG_DATA_HOME + # approach: /.helix_opencode_state/opencode/opencode.db. + # + # The sandbox branch is excluded because the sandbox applies the same + # knob itself, against a container path; see + # ``helix.sandbox._prepare_agent_state_dir``. opencode_state_dir = Path(worktree_path) / ".helix_opencode_state" - opencode_state_dir.mkdir(parents=True, exist_ok=True) - backend_env["XDG_DATA_HOME"] = str(opencode_state_dir) + (opencode_state_dir / backend).mkdir(parents=True, exist_ok=True) + backend_env.update( + agent_state_env(backend, state_root=str(opencode_state_dir)) + ) if sandbox is not None and sandbox.enabled: sandbox_image = resolve_sandbox_image(sandbox, backend) result = run_sandboxed_command( diff --git a/src/helix/sandbox.py b/src/helix/sandbox.py index 58a388f7..376aac9b 100644 --- a/src/helix/sandbox.py +++ b/src/helix/sandbox.py @@ -19,6 +19,11 @@ from pathlib import Path from typing import Literal +from helix.agent_state import ( + AGENT_STATE_CONTAINER_ROOT, + agent_state_env, + agent_state_subdirs, +) from helix.backends import BACKEND_AUTH_COMMANDS, DEFAULT_BACKEND_IMAGES from helix.config import EvaluatorSidecarConfig, SandboxConfig @@ -914,6 +919,33 @@ def sandbox_auth_volume_name(agent_backend: str) -> str: return f"helix-auth-{agent_backend}" +def _prepare_agent_state_dir( + tmp_path: Path, + *, + scope: Literal["agent", "evaluator"], + agent_backend: str | None, + image: str, +) -> Path | None: + """Create the per-candidate agent-state directory, or return ``None``. + + The directory lives inside the same temporary tree as the workspace copy, + so it inherits the sandbox's existing per-candidate scratch lifetime and is + removed by the ``_safe_rmtree`` in :func:`run_sandboxed_commands`. It is + chowned to ``node`` because the container runs as that user, matching how + the workspace copy is handed over. + """ + if scope != "agent" or agent_backend is None: + return None + subdirs = agent_state_subdirs(agent_backend) + if not subdirs: + return None + state_dir = tmp_path / "agent-state" + for name in subdirs: + (state_dir / name).mkdir(parents=True, exist_ok=True) + _docker_chown_workspace(state_dir, image, "node:node") + return state_dir + + def _docker_args( command: list[str], env: dict[str, str], @@ -924,6 +956,7 @@ def _docker_args( agent_backend: str | None, network: str | None = None, container_name: str | None = None, + agent_state_dir: Path | None = None, ) -> list[str]: args = [ "docker", @@ -946,6 +979,14 @@ def _docker_args( if agent_backend is None: raise ValueError("agent_backend is required for sandboxed agent commands") args.extend(["-v", f"{sandbox_auth_volume_name(agent_backend)}:/home/node:rw"]) + # The auth volume above is shared across candidates on purpose: it is + # what keeps token refresh and the CLIs' refresh locks working. The + # state mount below is per-candidate and lives outside /home/node, so + # nothing here changes how the credential is shared. + if agent_state_dir is not None: + args.extend( + ["-v", f"{agent_state_dir}:{AGENT_STATE_CONTAINER_ROOT}:rw"] + ) if sandbox.pids_limit is not None: args.extend(["--pids-limit", str(sandbox.pids_limit)]) @@ -967,6 +1008,10 @@ def _docker_args( container_env["PATH"] = ( "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ) + if agent_state_dir is not None and agent_backend is not None: + container_env.update( + agent_state_env(agent_backend, state_root=AGENT_STATE_CONTAINER_ROOT) + ) for key, value in container_env.items(): args.extend(["-e", f"{key}={value}"]) @@ -1013,6 +1058,12 @@ def run_sandboxed_commands( ) _init_synthetic_git_repo(workspace) _docker_chown_workspace(workspace, docker_image, "node:node") + agent_state_dir = _prepare_agent_state_dir( + tmp_path, + scope=scope, + agent_backend=agent_backend, + image=docker_image, + ) sidecar_runtime = ( current_evaluator_sidecar_runtime() if scope == "evaluator" else None ) @@ -1033,6 +1084,7 @@ def run_sandboxed_commands( agent_backend, sidecar_runtime.network if sidecar_runtime is not None else None, container_name=container_name, + agent_state_dir=agent_state_dir, ) try: results.append( diff --git a/tests/integration/__init__.py b/tests/integration/__init__.py new file mode 100644 index 00000000..e69de29b diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py new file mode 100644 index 00000000..49ba7197 --- /dev/null +++ b/tests/integration/conftest.py @@ -0,0 +1,130 @@ +"""Shared fixtures for container-backed integration tests. + +These tests run real backend containers. They never touch a real +``helix-auth-*`` login volume and never perform a login: every credential they +write is synthetic and lives in a throwaway volume that is removed again in +teardown. +""" + +from __future__ import annotations + +import os +import subprocess +import uuid +from collections.abc import Iterator + +import pytest + + +REAL_AUTH_VOLUME_PREFIX = "helix-auth-" +TEST_VOLUME_PREFIX = "helix-agent-state-test-" + + +def _strict() -> bool: + return os.environ.get("HELIX_DOCKER_TESTS_STRICT") == "1" + + +def _unavailable(reason: str) -> None: + """Fail under ``HELIX_DOCKER_TESTS_STRICT=1``, otherwise skip. + + Strict mode exists so CI cannot silently turn this suite into a no-op: + a missing daemon or image becomes a failure to fix rather than a green run. + """ + if _strict(): + pytest.fail(f"HELIX_DOCKER_TESTS_STRICT=1 but {reason}") + pytest.skip(reason) + + +def _docker(*args: str, check: bool = True) -> subprocess.CompletedProcess[str]: + return subprocess.run( + ["docker", *args], capture_output=True, text=True, check=check, timeout=300 + ) + + +@pytest.fixture(scope="session") +def docker_available() -> None: + try: + _docker("info") + except (OSError, subprocess.SubprocessError) as exc: + _unavailable(f"Docker daemon is not usable: {exc}") + + +@pytest.fixture +def require_image(docker_available: None): + def _require(image: str) -> str: + result = _docker("image", "inspect", image, check=False) + if result.returncode != 0: + _unavailable(f"image {image} is not present locally") + return image + + return _require + + +@pytest.fixture +def throwaway_volume(docker_available: None) -> Iterator[object]: + """Create synthetic-credential volumes and guarantee their removal. + + The name prefix is asserted on every call so a bug in a test can never + address, mutate, or delete one of the real ``helix-auth-*`` volumes. + """ + created: list[str] = [] + + def _create(image: str, seed_script: str) -> str: + name = f"{TEST_VOLUME_PREFIX}{uuid.uuid4().hex[:12]}" + assert not name.startswith(REAL_AUTH_VOLUME_PREFIX) + _docker("volume", "create", name) + created.append(name) + # Seed as root, then hand the tree to the container user the backends + # run as, mirroring how a real login volume ends up owned. + _docker( + "run", + "--rm", + "--network", + "none", + "-v", + f"{name}:/home/node", + "--user", + "root", + image, + "sh", + "-c", + f"set -eu; {seed_script}; chown -R node:node /home/node", + ) + return name + + try: + yield _create + finally: + for name in created: + assert name.startswith(TEST_VOLUME_PREFIX) + _docker("volume", "rm", "-f", name, check=False) + + +@pytest.fixture +def volume_listing(docker_available: None): + """Return names/modes/sizes of a volume's contents -- never file contents.""" + + def _list(volume: str, image: str) -> set[str]: + # ``~/.cache`` is pruned: the CLIs drop version-keyed caches there (for + # example cursor's V8 compile cache). Those are rebuilt from the image + # and carry no candidate state, so including them would make the + # comparison flap without telling us anything about contamination. + result = _docker( + "run", + "--rm", + "--network", + "none", + "-v", + f"{volume}:/home/node:ro", + "--user", + "node", + image, + "sh", + "-c", + 'find /home/node -mindepth 1 ' + '-not -path /home/node/.cache -not -path "/home/node/.cache/*" ' + '-printf "%M %s %p\\n" | sort -k3', + ) + return {line for line in result.stdout.splitlines() if line.strip()} + + return _list diff --git a/tests/integration/test_agent_state_isolation.py b/tests/integration/test_agent_state_isolation.py new file mode 100644 index 00000000..0c3bf1e2 --- /dev/null +++ b/tests/integration/test_agent_state_isolation.py @@ -0,0 +1,296 @@ +"""Container proof that agent state relocates and the credential does not. + +Each test asserts the same three things against a real backend container: + +(a) the backend's state lands in the per-candidate directory; +(b) nothing new lands in the shared login volume; +(c) the CLI still reports itself authenticated. + +Credentials are synthetic and live in throwaway volumes (see ``conftest``); +no test logs in, and none can reach a real ``helix-auth-*`` volume. +""" + +from __future__ import annotations + +import re +import subprocess +from pathlib import Path + +import pytest + +from helix.agent_state import ( + AGENT_STATE_CONTAINER_ROOT, + agent_state_cli_args, + agent_state_env, +) + + +pytestmark = pytest.mark.docker_integration + + +CODEX_IMAGE = "ghcr.io/ke7/helix-evo-runner-codex:latest" +CURSOR_IMAGE = "ghcr.io/ke7/helix-evo-runner-cursor:latest" +OPENCODE_IMAGE = "ghcr.io/ke7/helix-evo-runner-opencode:latest" + +# Deliberately malformed-but-well-shaped values. They are never accepted by a +# real API; they only have to be present for a CLI to report a stored login. +SYNTHETIC_CODEX_AUTH = ( + 'mkdir -p /home/node/.codex; printf "%s" ' + "'{\"OPENAI_API_KEY\":\"sk-SYNTHETIC-NOT-A-REAL-KEY\"}' " + "> /home/node/.codex/auth.json" +) +SYNTHETIC_CURSOR_AUTH = ( + 'mkdir -p /home/node/.config/cursor; printf "%s" ' + "'{\"accessToken\":\"SYNTHETIC\",\"refreshToken\":\"SYNTHETIC\"}' " + "> /home/node/.config/cursor/auth.json" +) +SYNTHETIC_OPENCODE_AUTH = ( + 'mkdir -p /home/node/.local/share/opencode; printf "%s" ' + "'{\"anthropic\":{\"type\":\"api\",\"key\":\"sk-ant-SYNTHETIC\"}}' " + "> /home/node/.local/share/opencode/auth.json" +) + + +def _run_backend( + *, + image: str, + volume: str, + state_dir: Path, + backend: str, + shell_command: str, + timeout: int = 120, +) -> str: + """Run one backend container the way ``helix.sandbox`` would. + + The auth volume is mounted read-write at ``/home/node`` exactly as in + production; the per-candidate state directory is a separate mount outside + it, carrying the relocation env vars from ``helix.agent_state``. + """ + (state_dir / backend).mkdir(parents=True, exist_ok=True) + args = [ + "docker", + "run", + "--rm", + "--network", + "none", + "--security-opt", + "no-new-privileges", + "--user", + "node", + "-v", + f"{volume}:/home/node:rw", + "-v", + f"{state_dir}:{AGENT_STATE_CONTAINER_ROOT}:rw", + "-e", + "HOME=/home/node", + ] + for key, value in agent_state_env( + backend, state_root=AGENT_STATE_CONTAINER_ROOT + ).items(): + args.extend(["-e", f"{key}={value}"]) + args.extend([image, "sh", "-lc", shell_command]) + result = subprocess.run( + args, capture_output=True, text=True, check=False, timeout=timeout + ) + return result.stdout + result.stderr + + +def _relative_paths(state_dir: Path) -> set[str]: + return { + str(p.relative_to(state_dir)) + for p in state_dir.rglob("*") + if p.is_file() + } + + +# --------------------------------------------------------------------------- +# codex +# --------------------------------------------------------------------------- + + +@pytest.mark.timeout(300) +def test_codex_state_databases_relocate( + tmp_path: Path, require_image, throwaway_volume, volume_listing +) -> None: + image = require_image(CODEX_IMAGE) + volume = throwaway_volume(image, SYNTHETIC_CODEX_AUTH) + before = volume_listing(volume, image) + + state_dir = tmp_path / "agent-state" + sqlite_args = " ".join( + agent_state_cli_args("codex", state_root=AGENT_STATE_CONTAINER_ROOT) + ) + _run_backend( + image=image, + volume=volume, + state_dir=state_dir, + backend="codex", + # The state databases are opened during startup, well before any API + # call, so a short timeout is enough and no model is ever reached + # (the container has no network). + shell_command=( + "cd /tmp; timeout 20 codex exec --json " + f"--dangerously-bypass-approvals-and-sandbox {sqlite_args} hi " + ">/dev/null 2>&1 || true" + ), + ) + + # (a) state landed per-candidate + relocated = _relative_paths(state_dir) + assert any(name.endswith("state_5.sqlite") for name in relocated), relocated + assert any(name.endswith("logs_2.sqlite") for name in relocated), relocated + + # (b) no sqlite state landed in the shared volume + after = volume_listing(volume, image) + new_entries = after - before + assert not [e for e in new_entries if ".sqlite" in e], new_entries + + # (c) the credential is untouched and still reported + status = _run_backend( + image=image, + volume=volume, + state_dir=state_dir, + backend="codex", + shell_command="codex login status", + ) + assert "Logged in" in status, status + + +# --------------------------------------------------------------------------- +# cursor +# --------------------------------------------------------------------------- + + +@pytest.mark.timeout(300) +def test_cursor_state_relocates_and_login_survives( + tmp_path: Path, require_image, throwaway_volume, volume_listing +) -> None: + image = require_image(CURSOR_IMAGE) + volume = throwaway_volume(image, SYNTHETIC_CURSOR_AUTH) + before = volume_listing(volume, image) + + state_dir = tmp_path / "agent-state" + status = _run_backend( + image=image, + volume=volume, + state_dir=state_dir, + backend="cursor", + shell_command="timeout 60 cursor-agent status", + ) + + # (a) ~/.cursor state landed per-candidate + assert "cursor/cli-config.json" in _relative_paths(state_dir) + + # (b) the shared volume is byte-for-byte unchanged + assert volume_listing(volume, image) == before + + # (c) the shared credential is still found + assert "Logged in" in status, status + + +def test_cursor_xdg_config_home_would_hide_the_credential( + tmp_path: Path, require_image, throwaway_volume +) -> None: + """Guard the rejected knob: XDG_CONFIG_HOME breaks cursor's login. + + This is why ``helix.agent_state`` uses CURSOR_CONFIG_DIR and why + ``cursor_credential_hazard`` warns when a user routes XDG_CONFIG_HOME + through ``passthrough_env``. + """ + image = require_image(CURSOR_IMAGE) + volume = throwaway_volume(image, SYNTHETIC_CURSOR_AUTH) + state_dir = tmp_path / "agent-state" + state_dir.mkdir() + + result = subprocess.run( + [ + "docker", "run", "--rm", "--network", "none", "--user", "node", + "-v", f"{volume}:/home/node:rw", + "-v", f"{state_dir}:{AGENT_STATE_CONTAINER_ROOT}:rw", + "-e", "HOME=/home/node", + "-e", f"XDG_CONFIG_HOME={AGENT_STATE_CONTAINER_ROOT}/cursor", + image, "sh", "-lc", "timeout 60 cursor-agent status", + ], + capture_output=True, text=True, check=False, timeout=120, + ) + assert "Not logged in" in result.stdout + result.stderr + + +# --------------------------------------------------------------------------- +# opencode +# --------------------------------------------------------------------------- + + +@pytest.mark.timeout(300) +def test_opencode_database_relocates_and_credential_stays( + tmp_path: Path, require_image, throwaway_volume, volume_listing +) -> None: + image = require_image(OPENCODE_IMAGE) + volume = throwaway_volume(image, SYNTHETIC_OPENCODE_AUTH) + before = volume_listing(volume, image) + + state_dir = tmp_path / "agent-state" + listing = _run_backend( + image=image, + volume=volume, + state_dir=state_dir, + backend="opencode", + shell_command="cd /tmp; timeout 120 opencode auth list", + timeout=200, + ) + + # (a) the database (which also carries token columns) landed per-candidate + assert "opencode/opencode.db" in _relative_paths(state_dir) + + # (b) no database landed in the shared volume + new_entries = volume_listing(volume, image) - before + assert not [e for e in new_entries if "opencode.db" in e], new_entries + + # (c) the shared auth.json is still the credential source, and is seen + assert "auth.json" in listing + assert "0 credentials" not in listing, listing + + +@pytest.mark.timeout(300) +def test_opencode_xdg_data_home_would_hide_the_credential( + tmp_path: Path, require_image, throwaway_volume +) -> None: + """Guard the rejected knob: XDG_DATA_HOME moves auth.json with the database.""" + image = require_image(OPENCODE_IMAGE) + volume = throwaway_volume(image, SYNTHETIC_OPENCODE_AUTH) + state_dir = tmp_path / "agent-state" + state_dir.mkdir() + + result = subprocess.run( + [ + "docker", "run", "--rm", "--network", "none", "--user", "node", + "-v", f"{volume}:/home/node:rw", + "-v", f"{state_dir}:{AGENT_STATE_CONTAINER_ROOT}:rw", + "-e", "HOME=/home/node", + "-e", f"XDG_DATA_HOME={AGENT_STATE_CONTAINER_ROOT}", + image, "sh", "-lc", "cd /tmp; timeout 120 opencode auth list", + ], + capture_output=True, text=True, check=False, timeout=200, + ) + assert "0 credentials" in result.stdout + result.stderr + + +# --------------------------------------------------------------------------- +# The invariant that outranks all of the above +# --------------------------------------------------------------------------- + + +def test_real_auth_volumes_are_never_addressed() -> None: + """No test in this suite may name a concrete login volume. + + Checked by inspecting the suite source rather than the daemon, so it holds + even on a machine that has no login volumes at all. Prose mentioning the + volume family in the abstract is fine; a resolvable name is not. + """ + concrete_name = re.compile(r"helix-auth-[a-z0-9]+") + for path in (Path(__file__), Path(__file__).parent / "conftest.py"): + for number, line in enumerate(path.read_text().splitlines(), start=1): + if concrete_name.search(line): + pytest.fail( + f"{path.name}:{number} names a real auth volume: {line.strip()}" + ) diff --git a/tests/unit/test_agent_state.py b/tests/unit/test_agent_state.py new file mode 100644 index 00000000..9eb66a6d --- /dev/null +++ b/tests/unit/test_agent_state.py @@ -0,0 +1,272 @@ +"""Tests: per-candidate agent-state relocation away from the shared auth volume. + +The invariant these tests defend is narrow and load-bearing: HELIX may move a +backend's *state* to a per-candidate location, but it must never move, copy, +name or shadow the *credential*, because the shared ``helix-auth-`` +volume is what keeps token refresh and the CLIs' refresh locks working. +""" + +from __future__ import annotations + +from pathlib import Path +from unittest.mock import MagicMock + +import pytest + +from helix.agent_state import ( + AGENT_STATE_CONTAINER_ROOT, + REJECTED_AGENT_STATE_KNOBS, + STATE_RELOCATING_BACKENDS, + UNRELOCATED_AGENT_STATE, + agent_state_cli_args, + agent_state_env, + agent_state_subdirs, + cursor_credential_hazard, +) +from helix.backends import BACKENDS +from helix.config import AgentConfig, SandboxConfig +from helix.mutator import _build_backend_args +from helix.sandbox import _docker_args, _prepare_agent_state_dir + + +# --------------------------------------------------------------------------- +# The state root must never sit inside the shared auth volume +# --------------------------------------------------------------------------- + + +def test_state_root_is_outside_the_auth_volume_mount() -> None: + """The per-candidate mount must not be nested under ``/home/node``. + + Mounting inside the auth volume would create a new entry in it, which is + exactly what the shared mount is not allowed to acquire. + """ + assert not AGENT_STATE_CONTAINER_ROOT.startswith("/home/node") + assert Path(AGENT_STATE_CONTAINER_ROOT).is_absolute() + + +# --------------------------------------------------------------------------- +# Per-backend knobs +# --------------------------------------------------------------------------- + + +def test_codex_relocates_state_databases_via_sqlite_home() -> None: + args = agent_state_cli_args("codex", state_root=AGENT_STATE_CONTAINER_ROOT) + assert args == ["-c", 'sqlite_home="/helix-state/codex"'] + + +def test_opencode_relocates_only_the_database_file() -> None: + env = agent_state_env("opencode", state_root=AGENT_STATE_CONTAINER_ROOT) + assert env == {"OPENCODE_DB": "/helix-state/opencode/opencode.db"} + + +def test_cursor_relocates_state_via_config_dir() -> None: + env = agent_state_env("cursor", state_root=AGENT_STATE_CONTAINER_ROOT) + assert env == {"CURSOR_CONFIG_DIR": "/helix-state/cursor"} + + +@pytest.mark.parametrize("backend", ["claude", "gemini"]) +def test_backends_without_a_safe_knob_get_nothing(backend: str) -> None: + """claude and gemini have no knob that separates state from credential.""" + assert agent_state_env(backend, state_root=AGENT_STATE_CONTAINER_ROOT) == {} + assert agent_state_cli_args(backend, state_root=AGENT_STATE_CONTAINER_ROOT) == [] + assert agent_state_subdirs(backend) == () + assert backend not in STATE_RELOCATING_BACKENDS + + +@pytest.mark.parametrize("backend", sorted(STATE_RELOCATING_BACKENDS)) +def test_relocating_backends_emit_exactly_one_knob(backend: str) -> None: + """Each backend uses one knob, so there is a single thing to re-verify.""" + knobs = list(agent_state_env(backend, state_root=AGENT_STATE_CONTAINER_ROOT)) + knobs += agent_state_cli_args(backend, state_root=AGENT_STATE_CONTAINER_ROOT)[:1] + assert len(knobs) == 1, f"{backend} should relocate state with one knob" + + +# --------------------------------------------------------------------------- +# Credential-safety: the knobs we must never emit +# --------------------------------------------------------------------------- + + +@pytest.mark.parametrize("backend", BACKENDS) +def test_no_backend_ever_receives_a_credential_moving_knob(backend: str) -> None: + """Regression guard for the knobs recorded in REJECTED_AGENT_STATE_KNOBS. + + Every name below relocates the backend's credential file along with its + state. Emitting any of them would make an existing login invisible to the + CLI, which is the failure this whole module exists to avoid. + """ + forbidden = { + "XDG_DATA_HOME", + "XDG_CONFIG_HOME", + "XDG_STATE_HOME", + "HOME", + "CODEX_HOME", + "CLAUDE_CONFIG_DIR", + "OPENCODE_CONFIG_DIR", + } + env = agent_state_env(backend, state_root=AGENT_STATE_CONTAINER_ROOT) + assert forbidden.isdisjoint(env), ( + f"{backend} must not receive a credential-relocating env var" + ) + rendered = " ".join( + agent_state_cli_args(backend, state_root=AGENT_STATE_CONTAINER_ROOT) + ) + assert "codex_home" not in rendered + + +def test_rejected_knobs_are_documented_with_a_reason() -> None: + """Keep the 'do not re-try this' list honest and non-empty.""" + assert REJECTED_AGENT_STATE_KNOBS + for name, reason in REJECTED_AGENT_STATE_KNOBS.items(): + assert ":" in name, f"{name} should read as 'backend:KNOB'" + assert reason.strip() + + +def test_every_backend_has_a_leftover_state_entry() -> None: + """Residue must be recorded for all backends, including the ones we fixed.""" + assert set(UNRELOCATED_AGENT_STATE) == set(BACKENDS) + + +# --------------------------------------------------------------------------- +# cursor's XDG_CONFIG_HOME hazard +# --------------------------------------------------------------------------- + + +def test_cursor_hazard_warns_when_xdg_config_home_is_present() -> None: + warning = cursor_credential_hazard("cursor", {"XDG_CONFIG_HOME": "/somewhere"}) + assert warning is not None + assert "XDG_CONFIG_HOME" in warning + + +def test_cursor_hazard_silent_when_absent_or_other_backend() -> None: + assert cursor_credential_hazard("cursor", {"PATH": "/usr/bin"}) is None + assert cursor_credential_hazard("codex", {"XDG_CONFIG_HOME": "/x"}) is None + + +# --------------------------------------------------------------------------- +# Sandbox wiring +# --------------------------------------------------------------------------- + + +def _agent_docker_args(backend: str, state_dir: Path | None) -> list[str]: + return _docker_args( + ["echo", "hi"], + {}, + Path("/tmp/workspace"), + SandboxConfig(enabled=True), + "agent", + "img:latest", + backend, + agent_state_dir=state_dir, + ) + + +def test_shared_auth_volume_mount_is_unchanged_by_relocation() -> None: + """The credential mount must stay ``:/home/node:rw``, always.""" + for backend in BACKENDS: + with_state = _agent_docker_args(backend, Path("/tmp/state")) + without_state = _agent_docker_args(backend, None) + expected = f"helix-auth-{backend}:/home/node:rw" + assert expected in with_state + assert expected in without_state + # Relocation adds mounts; it never removes or rewrites the auth mount. + assert with_state.count(expected) == without_state.count(expected) == 1 + + +def test_state_dir_is_mounted_outside_the_auth_volume() -> None: + args = _agent_docker_args("codex", Path("/tmp/state")) + assert f"/tmp/state:{AGENT_STATE_CONTAINER_ROOT}:rw" in args + # No mount target may be nested inside the shared volume. + targets = [ + args[i + 1].split(":")[1] for i, a in enumerate(args) if a == "-v" + ] + nested = [t for t in targets if t.startswith("/home/node/")] + assert not nested, f"mounts nested inside the auth volume: {nested}" + + +def test_relocation_env_reaches_the_container() -> None: + args = _agent_docker_args("cursor", Path("/tmp/state")) + assert f"CURSOR_CONFIG_DIR={AGENT_STATE_CONTAINER_ROOT}/cursor" in args + args = _agent_docker_args("opencode", Path("/tmp/state")) + assert f"OPENCODE_DB={AGENT_STATE_CONTAINER_ROOT}/opencode/opencode.db" in args + + +def test_evaluator_scope_gets_no_state_dir(tmp_path: Path) -> None: + """Only agent commands touch the auth volume, so only they need relocation.""" + assert ( + _prepare_agent_state_dir( + tmp_path, scope="evaluator", agent_backend="codex", image="img" + ) + is None + ) + + +def test_no_state_dir_for_backends_without_a_knob(tmp_path: Path) -> None: + assert ( + _prepare_agent_state_dir( + tmp_path, scope="agent", agent_backend="claude", image="img" + ) + is None + ) + + +def test_state_dir_lives_in_the_per_candidate_scratch_tree( + tmp_path: Path, mocker +) -> None: + """The directory must sit under the sandbox temp tree that is rmtree'd.""" + mocker.patch("helix.sandbox._docker_chown_workspace") + state_dir = _prepare_agent_state_dir( + tmp_path, scope="agent", agent_backend="codex", image="img" + ) + assert state_dir is not None + assert state_dir.is_relative_to(tmp_path) + assert (state_dir / "codex").is_dir() + + +# --------------------------------------------------------------------------- +# Backend argv wiring +# --------------------------------------------------------------------------- + + +def test_codex_argv_carries_sqlite_home_when_sandboxed() -> None: + args = _build_backend_args( + "/workspace", + AgentConfig(backend="codex"), + "prompt.md", + agent_state_root=AGENT_STATE_CONTAINER_ROOT, + ) + assert "-c" in args + assert 'sqlite_home="/helix-state/codex"' in args + + +def test_codex_argv_unchanged_without_a_sandbox() -> None: + """Unsandboxed runs have no container state mount to point at.""" + args = _build_backend_args("/wt", AgentConfig(backend="codex"), "prompt.md") + assert not any("sqlite_home" in a for a in args) + + +@pytest.mark.parametrize("backend", ["claude", "cursor", "gemini", "opencode"]) +def test_non_codex_argv_never_carries_a_state_override(backend: str) -> None: + args = _build_backend_args( + "/workspace", + AgentConfig(backend=backend), + "prompt.md", + agent_state_root=AGENT_STATE_CONTAINER_ROOT, + ) + assert not any("sqlite_home" in a for a in args) + + +def test_local_opencode_db_stays_in_the_gitignored_state_dir( + tmp_path: Path, mocker +) -> None: + """Unsandboxed opencode keeps its database inside .helix_opencode_state/.""" + from helix.mutator import invoke_claude_code + + mock_run = mocker.patch("helix.mutator.subprocess.run") + mock_run.return_value = MagicMock( + stdout='{"type":"result","sessionID":"ses_abc"}\n', stderr="", returncode=0 + ) + invoke_claude_code(str(tmp_path), "prompt", AgentConfig(backend="opencode")) + + db_path = Path(mock_run.call_args[1]["env"]["OPENCODE_DB"]) + assert db_path.is_relative_to(tmp_path / ".helix_opencode_state") + assert db_path.parent.is_dir(), "parent dir must exist before opencode starts" diff --git a/tests/unit/test_mutator.py b/tests/unit/test_mutator.py index ff985953..27ba9031 100644 --- a/tests/unit/test_mutator.py +++ b/tests/unit/test_mutator.py @@ -1873,18 +1873,22 @@ def test_claude_tool_counts_patched_from_transcript( # --------------------------------------------------------------------------- -# Tests: OpenCode per-candidate SQLite isolation (XDG_DATA_HOME) +# Tests: OpenCode per-candidate SQLite isolation (OPENCODE_DB) # --------------------------------------------------------------------------- class TestOpenCodeSubprocessIsolation: - """Verify that concurrent opencode subprocesses receive isolated XDG_DATA_HOME + """Verify that concurrent opencode subprocesses receive isolated OPENCODE_DB values so each worker opens its own SQLite database and the shared-database 'PRAGMA journal_mode = WAL' contention observed in PR #34 cannot recur. + + OPENCODE_DB rather than XDG_DATA_HOME: the latter relocates opencode's + ``auth.json`` along with the database, which makes an existing login + invisible to the CLI. """ def test_opencode_subprocess_isolation_env_set(self, tmp_path: Path, mocker): - """invoke_claude_code sets XDG_DATA_HOME for opencode backend.""" + """invoke_claude_code sets OPENCODE_DB for opencode backend.""" mock_run = mocker.patch("helix.mutator.subprocess.run") mock_run.return_value = MagicMock( stdout='{"type":"result","sessionID":"ses_abc"}\n', @@ -1898,20 +1902,25 @@ def test_opencode_subprocess_isolation_env_set(self, tmp_path: Path, mocker): call_kwargs = mock_run.call_args[1] env = call_kwargs["env"] - assert "XDG_DATA_HOME" in env, ( - "XDG_DATA_HOME must be set for opencode to isolate its SQLite database " + assert "OPENCODE_DB" in env, ( + "OPENCODE_DB must be set for opencode to isolate its SQLite database " "from other concurrent opencode workers" ) # Must point inside the candidate worktree so cleanup is automatic - xdg = env["XDG_DATA_HOME"] - assert xdg.startswith(str(tmp_path)), ( - f"XDG_DATA_HOME={xdg!r} should be under the candidate worktree {tmp_path}" + db_path = env["OPENCODE_DB"] + assert db_path.startswith(str(tmp_path)), ( + f"OPENCODE_DB={db_path!r} should be under the candidate worktree {tmp_path}" + ) + # The credential knob must stay untouched: relocating XDG_DATA_HOME + # would move auth.json with the database and hide an existing login. + assert "XDG_DATA_HOME" not in env, ( + "XDG_DATA_HOME must not be set for opencode; it moves auth.json too" ) def test_opencode_subprocess_isolation_unique_per_candidate( self, tmp_path: Path, mocker ): - """Two different candidates get different XDG_DATA_HOME values.""" + """Two different candidates get different OPENCODE_DB values.""" mock_run = mocker.patch("helix.mutator.subprocess.run") mock_run.return_value = MagicMock( stdout='{"type":"result","sessionID":"ses_abc"}\n', @@ -1925,18 +1934,18 @@ def test_opencode_subprocess_isolation_unique_per_candidate( wt_b.mkdir() invoke_claude_code(str(wt_a), "prompt", AgentConfig(backend="opencode")) - env_a = mock_run.call_args[1]["env"]["XDG_DATA_HOME"] + env_a = mock_run.call_args[1]["env"]["OPENCODE_DB"] invoke_claude_code(str(wt_b), "prompt", AgentConfig(backend="opencode")) - env_b = mock_run.call_args[1]["env"]["XDG_DATA_HOME"] + env_b = mock_run.call_args[1]["env"]["OPENCODE_DB"] assert env_a != env_b, ( - "Each candidate worktree must produce a distinct XDG_DATA_HOME so their " + "Each candidate worktree must produce a distinct OPENCODE_DB so their " "opencode SQLite databases don't collide" ) def test_opencode_subprocess_inherits_other_env(self, tmp_path: Path, mocker): - """Non-isolation env vars (PATH, HOME) are still present after XDG injection.""" + """Non-isolation env vars (PATH, HOME) survive the OPENCODE_DB injection.""" mock_run = mocker.patch("helix.mutator.subprocess.run") mock_run.return_value = MagicMock( stdout='{"type":"result","sessionID":"ses_abc"}\n', @@ -1956,13 +1965,13 @@ def test_opencode_subprocess_inherits_other_env(self, tmp_path: Path, mocker): assert "PATH" in env, "PATH must survive the env scrub for opencode" if "HOME" in os.environ: assert "HOME" in env, "HOME must survive the env scrub for opencode" - # And XDG_DATA_HOME is the *only* new opencode-specific addition - assert "XDG_DATA_HOME" in env + # And OPENCODE_DB is the *only* new opencode-specific addition + assert "OPENCODE_DB" in env def test_opencode_isolation_not_applied_to_other_backends( self, tmp_path: Path, mocker ): - """XDG_DATA_HOME must NOT be injected for claude/codex/cursor/gemini.""" + """OPENCODE_DB must NOT be injected for claude/codex/cursor/gemini.""" mock_run = mocker.patch("helix.mutator.subprocess.run") mock_run.return_value = MagicMock(stdout="{}", stderr="", returncode=0) @@ -1971,8 +1980,8 @@ def test_opencode_isolation_not_applied_to_other_backends( str(tmp_path), "prompt", AgentConfig(backend=backend) ) env = mock_run.call_args[1]["env"] - assert "XDG_DATA_HOME" not in env, ( - f"XDG_DATA_HOME must not be injected for {backend} backend" + assert "OPENCODE_DB" not in env, ( + f"OPENCODE_DB must not be injected for {backend} backend" ) def test_opencode_isolation_dir_gitignored(self, tmp_path: Path, mocker): diff --git a/tests/unit/test_sandbox.py b/tests/unit/test_sandbox.py index 9bdf9923..210aa9b8 100644 --- a/tests/unit/test_sandbox.py +++ b/tests/unit/test_sandbox.py @@ -136,9 +136,18 @@ def fake_run(args, **kwargs): assert "helix-auth-codex:/home/node:rw" in docker_call assert f"{tmp_path}:" not in joined assert "/workspace:rw" in joined + # codex relocates its state databases, so the agent container also gets a + # per-candidate state mount -- outside /home/node, leaving the shared auth + # mount asserted above untouched. + assert "/helix-state:rw" in joined + assert "/home/node/helix-state" not in joined + # Three housekeeping chowns: the workspace before the run, the + # per-candidate state directory before the run (it must be writable by the + # container's ``node`` user), and the workspace again afterwards. chown_calls = [call for call in calls if _is_workspace_chown(call)] - assert len(chown_calls) == 2 + assert len(chown_calls) == 3 assert "node:node" in chown_calls[0] + assert "node:node" in chown_calls[1] def test_evaluator_scope_does_not_mount_agent_auth(tmp_path: Path, mocker): From 3af85bd73b2f52337cdaadce770f86722ada9189 Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Wed, 26 Aug 2026 20:21:22 +0300 Subject: [PATCH 02/16] feat(evolution): warm the shared credential once per generation, and name credential failures Candidates share one login volume read-write, and that stays. What is not safe is the first moment after the shared credential goes stale: every candidate in the generation decides independently that a refresh is due and posts the same single-use refresh token. Measured against the shipped codex CLI with a synthetic credential in a throwaway volume and a local single-use token endpoint: five simultaneous candidates produce five exchanges, one grant, and four "your refresh token was already used" -- and all five exit 0 with empty stderr. A whole generation can die without saying anything. Today the only thing standing between a run and that outcome is codex's own last_refresh field happening to keep the window narrow. Warm the credential once per generation, before any candidate is dispatched, through the sandboxed auth container that already exists and is already a single writer. Same measurement with the warm in front: one exchange, granted, and the five candidates then perform none, because there is nothing left for them to refresh. Per generation and not per run: a run outlives any refresh interval, so a single warm at startup stops protecting it the moment the credential next goes stale mid-flight. Only codex is warmed, and NOT with its registered status command: codex `codex login status` never takes the refresh path -- it exits 0 without issuing a request whether last_refresh is minutes or 30 days old, so warming with it would be a placebo. The warm runs `codex debug models`, which loads auth through the refreshing path and is free: it completes with --network none on a fresh credential, writes nothing, and its only request when a refresh IS due is the OAuth token exchange. No model is invoked either way. That "free" check is the constraint that decides whether a warm may ship at all -- this runs on the operator's paid account every generation. claude skipped: takes a real cross-process lock, retries, re-reads. cursor skipped: never spends its stored refresh token. opencode skipped: refreshes only from inside the fetch wrapper that issues a model request, so no free command performs it. `opencode providers list` was measured free -- and for the same reason refreshes nothing. gemini skipped: no free command known to take the refresh path, and no credential to measure one against. Not warmed on a guess. Each reason is recorded in CREDENTIAL_WARM_SKIP_REASONS, and a test asserts every backend is either warmed or explained, so a new backend cannot fall through silently. A failed warm is logged and the run continues -- candidates may still work on the credential already stored. Second half: make the failure visible. There was no auth-failure detection at all, and the structured result envelope's is_error field was never read anywhere in src/, so an unusable login was indistinguishable from an agent that wrote bad code. Classify credential exhaustion as its own failure kind, anchored on wording read out of the shipped binaries rather than guessed ("Your access token could not be refreshed" and its four suffixes from codex, "Token refresh failed:" from opencode, "User OAuth refresh failed" from claude). is_error narrows what is scanned but never classifies on its own: a tool_result carrying is_error is usually the agent's own failing command, and a candidate whose work happens to be about refresh tokens must never be able to report the operator's login as broken. On a zero exit only is_error-flagged envelope text is considered for that reason; on a non-zero exit the raw streams are read too. The result surfaces where an operator sees it: the slot says the login failed rather than the code, and the permanent end-of-run summary repeats it after the live display is gone. No config knob: a credential that cannot be refreshed is never the behaviour anyone wants, and a warm that costs nothing needs no opt-out. --- src/helix/backends.py | 97 +++++++ src/helix/evolution.py | 149 +++++++++- src/helix/exceptions.py | 23 ++ src/helix/mutator.py | 198 +++++++++++++ src/helix/sandbox.py | 97 ++++++- .../test_credential_warm_docker.py | 204 +++++++++++++ .../test_credential_failure_classification.py | 274 ++++++++++++++++++ tests/unit/test_credential_warm.py | 273 +++++++++++++++++ tests/unit/test_credential_warm_loop.py | 193 ++++++++++++ 9 files changed, 1503 insertions(+), 5 deletions(-) create mode 100644 tests/integration/test_credential_warm_docker.py create mode 100644 tests/unit/test_credential_failure_classification.py create mode 100644 tests/unit/test_credential_warm.py create mode 100644 tests/unit/test_credential_warm_loop.py diff --git a/src/helix/backends.py b/src/helix/backends.py index 4f5fe838..9e5d15a2 100644 --- a/src/helix/backends.py +++ b/src/helix/backends.py @@ -89,6 +89,28 @@ "login": ["codex", "login", "--device-auth"], "status": ["codex", "login", "status"], "logout": ["codex", "logout"], + # Credential warm -- see CREDENTIAL_WARM_SKIP_REASONS below for why + # codex is the only backend with one. + # + # NOT ``codex login status``. Measured against codex-cli 0.130.0 with + # a synthetic credential in a throwaway volume: ``codex login status`` + # prints "Logged in using ChatGPT" and exits 0 without issuing a single + # request, whether the stored ``last_refresh`` is minutes or 30 days + # old. It reads auth.json; it never takes the refresh path, so warming + # with it would be a placebo. + # + # ``codex debug models`` renders the CLI's built-in model catalog. It + # loads auth through the refreshing path, so it performs the refresh + # this warm exists to perform, and it is free: + # * with a fresh credential it completes with ``--network none``, + # writes nothing to the login volume, and makes no request at all; + # * with a stale credential its only request is the OAuth token + # exchange, which the refreshed credential is then written back + # from. No model is invoked and no quota is consumed either way. + # stdout is discarded because the catalog is ~200 KB and the command is + # run for its side effect on the credential, not for its output; + # stderr is kept so a failure stays diagnosable. + "warm": ["sh", "-lc", "set -eu; codex debug models >/dev/null"], }, "cursor": { "login": ["cursor-agent", "login"], @@ -115,5 +137,80 @@ } +# --------------------------------------------------------------------------- +# Per-generation credential warm +# --------------------------------------------------------------------------- +# +# Every candidate container mounts the shared login volume read-write, which is +# what lets a backend CLI refresh its own OAuth token and keep the refreshed +# credential for the next candidate. The hazard is the *first* moment after a +# credential goes stale: several candidates start at once, each decides +# independently that a refresh is due, and each posts the same single-use +# refresh token. One wins; the rest are told the token was already consumed. +# +# ``helix.sandbox.warm_backend_credential`` closes that window by running the +# command below once, in one container, before a generation dispatches any +# candidate -- so whatever refresh is due happens under a single writer and +# every candidate then starts from an already-fresh credential with nothing +# left to race for. +# +# A backend is warmed only when a command exists here that (a) actually takes +# the CLI's refresh path and (b) costs nothing. Both halves are load-bearing: +# a command that never refreshes buys no safety, and a command that bills the +# operator's account once per generation would be worse than the race it +# prevents. Backends with no entry are listed in +# CREDENTIAL_WARM_SKIP_REASONS with the reason they need none. + + +CREDENTIAL_WARM_SKIP_REASONS: dict[str, str] = { + "claude": ( + "Claude Code serialises its own refresh: it takes a real cross-process " + "lock file, retries while another process holds it, and re-reads the " + "credential afterwards, so concurrent candidates cannot consume the " + "same refresh token. Warming would add a container per generation and " + "remove no hazard." + ), + "cursor": ( + "Cursor Agent never spends its stored refresh token: it re-exchanges " + "an API key instead, so there is no single-use grant for candidates to " + "compete over." + ), + "gemini": ( + "No free Gemini CLI command is known to take the refresh path. The " + "registered status command is `gemini --version`, which reports the " + "version and touches no credential, so warming with it would be a " + "placebo; and no Gemini credential exists to measure a real refresh " + "against. Left unwarmed deliberately rather than warmed on a guess." + ), + "opencode": ( + "OpenCode refreshes an `oauth`-type credential only from inside the " + "fetch wrapper that issues a model request -- read from opencode-ai " + "1.14.24, which refreshes when `expires` has passed and writes the new " + "credential back unlocked. There is therefore no command that performs " + "that refresh without also invoking a model, and a per-generation model " + "call on the operator's account is a worse cost than the race. " + "`opencode providers list` was measured to be free -- it completes with " + "`--network none` against an expired oauth credential and leaves " + "auth.json byte-identical -- but for exactly that reason it refreshes " + "nothing. `api`-type credentials never refresh and are not at risk." + ), +} +"""Why a backend has no ``warm`` entry in :data:`BACKEND_AUTH_COMMANDS`. + +Skipping is a correctness statement, not an optimisation: each entry records +either that the backend cannot lose a refresh race, or that no free command +would win it. +""" + + +def backend_credential_warm_skip_reason(backend: str) -> str | None: + """Return why *backend* is not credential-warmed, or ``None`` if it is.""" + if "warm" in BACKEND_AUTH_COMMANDS.get(backend, {}): + return None + return CREDENTIAL_WARM_SKIP_REASONS.get( + backend, "no credential-warm command is registered for this backend" + ) + + def backend_display_name(backend: str) -> str: return BACKEND_DISPLAY_NAMES.get(backend, backend) diff --git a/src/helix/evolution.py b/src/helix/evolution.py index 147db9ae..44d7e03c 100644 --- a/src/helix/evolution.py +++ b/src/helix/evolution.py @@ -14,7 +14,7 @@ import traceback from collections.abc import Callable from concurrent.futures import ThreadPoolExecutor, as_completed -from dataclasses import replace +from dataclasses import dataclass, field, replace from pathlib import Path from typing import Any @@ -47,7 +47,9 @@ set_phase, ) +from helix.backends import backend_display_name from helix.exceptions import ( + CredentialRefreshError, HelixError, PromptArtifactCollisionError, RateLimitError, @@ -77,7 +79,11 @@ HelixResult, ParetoFrontier, ) -from helix.sandbox import start_evaluator_sidecar +from helix.sandbox import ( + CredentialWarmResult, + start_evaluator_sidecar, + warm_backend_credential, +) from helix.state import ( BudgetState, clear_eval_cache, @@ -1446,6 +1452,95 @@ def _plan_proposals( # upstream's ``ReflectiveMutationProposer.propose``, though upstream # batches these stages across all sampled tasks per iteration instead # of running one call per proposal slot. +@dataclass +class CredentialFailureLog: + """Every credential failure seen during a run, in the order observed. + + Proposal workers run in a thread pool, so the list is appended under a + lock. The log exists so a run that dies from an unusable login says so + once, plainly, in the permanent end-of-run summary -- not only in a + per-slot error that has already scrolled past by the time the run ends. + """ + + entries: list[tuple[str, str]] = field(default_factory=list) + _lock: threading.Lock = field(default_factory=threading.Lock, repr=False) + + def record(self, candidate_id: str, message: str) -> None: + with self._lock: + self.entries.append((candidate_id, message)) + + def __len__(self) -> int: + with self._lock: + return len(self.entries) + + def candidate_ids(self) -> list[str]: + with self._lock: + return [candidate_id for candidate_id, _ in self.entries] + + def last_message(self) -> str: + with self._lock: + return self.entries[-1][1] if self.entries else "" + + +def _warm_generation_credential( + config: HelixConfig, *, gen: int, announce_skip: bool +) -> CredentialWarmResult | None: + """Refresh the agent backend's shared credential once for generation *gen*. + + Called once per generation, before any candidate is dispatched, so that a + refresh which has come due happens under one writer and every candidate in + the generation then starts from an already-fresh credential. + + Per *generation* rather than once per run on purpose: a long run outlives + any refresh interval, so a single warm at startup stops protecting the run + the moment the credential next goes stale mid-flight. + + Returns ``None`` when there is nothing to warm -- an unsandboxed run has no + HELIX-managed login volume, because the backend runs directly against the + operator's own CLI state and HELIX never mounts or arbitrates it. + + Never fatal. A warm that could not run leaves exactly today's behaviour in + place (candidates refresh for themselves and may race), and candidates may + still succeed on the credential already stored -- so the run continues and + the operator is told, in those terms, what protection was lost. + """ + if not config.sandbox.enabled: + return None + + backend = config.agent.backend + display = backend_display_name(backend) + result = warm_backend_credential(backend, sandbox=config.sandbox) + + if result.skipped: + # The reason is a property of the backend, not of this generation, so + # say it once per run instead of once per generation. + if announce_skip: + logger.info( + "No credential warm for %s: %s", display, result.skip_reason + ) + return result + + if result.warmed: + logger.debug( + "Credential warm for %s completed before generation %d.", display, gen + ) + return result + + detail = f" Detail: {result.detail}" if result.detail else "" + message = ( + f"Credential warm for {display} did not complete before generation " + f"{gen} (exit {result.returncode}). Candidates in this generation will " + "each decide for themselves whether to refresh the shared login, and " + "if a refresh is due they can spend the same single-use refresh token " + "at once -- the losers of that race can fail without reporting an " + "error of their own. The run continues: the credential already stored " + f"may still be usable.{detail}" + ) + logger.warning("%s", message) + print_warning(message) + return result + + def _run_proposal_worker( pre_ctx: ProposalContext, *, @@ -1457,6 +1552,7 @@ def _run_proposal_worker( evaluator_manifest: dict[str, str], use_minibatch_gate: bool, gen: int, + credential_failures: CredentialFailureLog, ) -> ProposalResult: """Atomic proposal worker — mirrors the sample/evaluate/mutate/evaluate shape of GEPA's ``ReflectiveMutationProposer.propose``. @@ -1579,6 +1675,23 @@ def _run_proposal_worker( f"retries — proposal slot skipped. " f"Run [cyan]helix resume[/cyan] when rate limits clear." ) + elif isinstance(_mu_exc, CredentialRefreshError): + # Name the failure for what it is. Without this the slot is + # indistinguishable from a mutation that produced bad code, + # and a whole generation can die quietly on a broken login. + credential_failures.record(_new_id, str(_mu_exc)) + logger.error( + "Mutation %s (parent: %s, gen %d) failed on the shared " + "%s credential, not on its code: %s", + _new_id, _parent.id, gen, + backend_display_name(config.agent.backend), _mu_exc, + ) + print_error( + f"Mutation [bold]{_new_id}[/bold] failed because the shared " + f"{backend_display_name(config.agent.backend)} credential " + f"could not be used or refreshed — this is a login failure, " + f"not a failure of the candidate's code." + ) else: print_error( f"Parallel mutation {_new_id} (parent: {_parent.id}, gen {gen}) " @@ -2186,6 +2299,10 @@ def _sync_frontier_state() -> None: # Mutation counters for display mutations_attempted = 0 mutations_accepted = 0 + credential_failures = CredentialFailureLog() + # The credential warm's skip reason is a fact about the backend, not + # about any one generation; announce it once. + credential_warm_skip_announced = False gen = start_gen - 1 while gen < config.evolution.max_generations: @@ -2219,6 +2336,17 @@ def _sync_frontier_state() -> None: print_warning("Budget exhausted -- stopping early.") break + # ---- Credential warm (once per generation) ------------------- + # Sits above the merge/mutate split so it covers every path that + # dispatches a candidate this generation, and above every + # candidate so the shared login is already fresh by the time any + # of them could start refreshing it themselves. + _warm = _warm_generation_credential( + config, gen=gen, announce_skip=not credential_warm_skip_announced + ) + if _warm is not None and _warm.skipped: + credential_warm_skip_announced = True + # ============================================================= # GEPA parity (Fix 6/7): Merge OR mutate per iteration. # Merge fires FIRST at the start of the iteration (deferred from @@ -2744,6 +2872,7 @@ def _has_val_support_overlap(i: str, j: str) -> bool: evaluator_manifest=evaluator_manifest, use_minibatch_gate=use_minibatch_gate, gen=gen, + credential_failures=credential_failures, ), max_workers=config.evolution.max_workers, gen=gen, @@ -3441,6 +3570,22 @@ def _drop_duplicate_child(gated: GatedProposal) -> bool: render_budget(state.budget, config.evolution) render_frontier_table(frontier, frontier._results) + # A credential failure is not a code failure, and the operator has to be + # able to tell them apart after the fact. The per-slot errors above have + # long scrolled away by now; this line is part of the permanent summary + # that outlives the live display. + if credential_failures: + _failed_ids = ", ".join(credential_failures.candidate_ids()) + print_error( + f"{len(credential_failures)} mutation(s) failed on the shared " + f"{backend_display_name(config.agent.backend)} credential, not on " + f"their code: {_failed_ids}. The backend reported that its stored " + f"login could not be used or refreshed. Re-authenticate with " + f"[cyan]helix sandbox login {config.agent.backend}[/cyan], then " + f"[cyan]helix resume[/cyan]. Last report: " + f"{credential_failures.last_message()}" + ) + best = frontier.best() print_success(f"Evolution complete. Best candidate: {best.id}") diff --git a/src/helix/exceptions.py b/src/helix/exceptions.py index 8f733094..198fb234 100644 --- a/src/helix/exceptions.py +++ b/src/helix/exceptions.py @@ -121,6 +121,29 @@ class RateLimitError(HelixError): """ +class CredentialRefreshError(HelixError): + """Raised when a backend CLI could not use or refresh its stored credential. + + This is a *credential* failure, not a code failure and not a quota failure. + It fires when the agent CLI reports that its OAuth refresh token was + already used, expired, revoked, or is missing -- the state HELIX reaches + when several candidates sharing one login volume all try to spend the same + single-use refresh token at once, or when the operator's login has simply + lapsed. + + It exists so a run that dies this way says so. Without it the failure is + indistinguishable from "the agent wrote bad code": the mutation is + abandoned, the proposal slot is dropped, and the operator sees a generation + that produced nothing with no reason attached. + + Inherits from :class:`HelixError` (not :class:`MutationError`) so the + proposal worker in ``evolution.py`` can route it separately -- the run + continues, but every credential-classified failure is counted and named in + the end-of-run summary. Detection is anchored on distinctive wording read + out of the shipped backend CLIs; see ``helix.mutator``. + """ + + # --------------------------------------------------------------------------- # Formatted error printing # --------------------------------------------------------------------------- diff --git a/src/helix/mutator.py b/src/helix/mutator.py index e063fd0b..493fe34b 100644 --- a/src/helix/mutator.py +++ b/src/helix/mutator.py @@ -21,6 +21,7 @@ from helix.population import Candidate, EvalResult from helix.config import AgentConfig, HelixConfig, SandboxConfig from helix.exceptions import ( + CredentialRefreshError, MutationError, PromptArtifactCollisionError, RateLimitError, @@ -602,6 +603,145 @@ def _looks_like_rate_limit(text: str) -> bool: return any(kw in lower for kw in _RATE_LIMIT_KEYWORDS) +# --------------------------------------------------------------------------- +# Credential / refresh-exhaustion detection +# --------------------------------------------------------------------------- +# +# Distinct from the rate-limit detection above. A rate limit means "come back +# later on the same credential"; these markers mean "the credential itself can +# no longer be used", which is what a lost refresh race looks like from inside +# a candidate. Telling them apart matters because only the second one is +# unrecoverable without operator action, and neither is a code failure. +# +# Every marker below is a phrase read out of the shipped CLI it belongs to, not +# a guess at the wording. They are deliberately whole distinctive sentences or +# clauses: matching a bare number or a single common word ("401", "token", +# "auth") is the false-positive trap this repo has already paid for once -- +# a candidate whose own diff or test output mentions tokens must never be +# reported to the operator as a broken login. +_CREDENTIAL_FAILURE_MARKERS: tuple[str, ...] = ( + # Codex CLI (codex-cli 0.130.0). One prefix covers every suffix the CLI + # appends: "... because your refresh token was already used." / "... has + # expired." / "... was revoked." / "... because you have since logged out + # or signed in to another account." / the bare + # "Your access token could not be refreshed. Please log out and sign in + # again." The already-used variant is the one a lost refresh race + # produces. + "your access token could not be refreshed", + "failed to refresh token while getting account", + "chatgpt account id not available, please re-run `codex login`", + # OpenCode (opencode-ai 1.14.24) -- thrown by the provider fetch wrapper + # when the OAuth token exchange is rejected, e.g. "Token refresh failed: + # 400". The colon is kept so the phrase cannot match narrative prose. + "token refresh failed:", + # Claude Code (2.1.138). + "user oauth refresh failed", + "api error: 401 invalid api key", +) + + +def credential_failure_marker(text: str) -> str | None: + """Return the credential-failure marker *text* contains, or ``None``. + + Returns the matched marker rather than a bool so callers can name the + evidence in the operator-facing message instead of asserting a verdict + with nothing behind it. + """ + if not text: + return None + lower = text.lower() + for marker in _CREDENTIAL_FAILURE_MARKERS: + if marker in lower: + return marker + return None + + +def _errored_envelope_texts(parsed: dict[str, Any]) -> list[str]: + """Return the message text of every envelope node flagged ``is_error``. + + ``is_error`` is the backends' own "this node reports a failure" flag: it is + top-level on Claude Code's JSON result envelope and per-event inside the + JSONL streams the other backends emit. Reading it is what lets a + credential failure be recognised on a *zero* exit, where there is no exit + code to key off. + + The flag alone never classifies anything. A ``tool_result`` carrying + ``is_error`` is usually just the agent's own failing shell command -- an + ordinary code failure. It only narrows the text that + :func:`credential_failure_marker` is then asked about, so a classification + still needs the backend to have said, in its own words, that the + credential is unusable. + """ + texts: list[str] = [] + for node in _walk_json(parsed): + flag = node.get("is_error") + if flag is not True: + continue + for key in ("error", "message", "result", "error_message", "text", "content"): + value = node.get(key) + if isinstance(value, str) and value: + texts.append(value) + return texts + + +def _credential_failure_evidence( + parsed: dict[str, Any] | None, + result: subprocess.CompletedProcess[str], +) -> tuple[str, str] | None: + """Return ``(marker, where)`` when this invocation is a credential failure. + + On a **zero** exit only ``is_error``-flagged envelope text is considered. + The run reported success, so the raw streams are full of the agent's own + work; scanning them would let a candidate that merely *edited* an OAuth + code path talk HELIX into declaring the operator's login broken. + + On a **non-zero** exit the raw streams are read too. The invocation has + already failed, so the only question left is which kind of failure it was, + and CLIs routinely report an unusable credential on stderr without ever + emitting a structured event. + """ + for text in _errored_envelope_texts(parsed or {}): + marker = credential_failure_marker(text) + if marker is not None: + return marker, "structured result envelope (is_error)" + if result.returncode == 0: + return None + for where, text in (("stderr", result.stderr), ("stdout", result.stdout)): + marker = credential_failure_marker(text or "") + if marker is not None: + return marker, where + return None + + +def _credential_refresh_error( + *, + backend: str, + backend_name: str, + marker: str, + where: str, + cmd_str: str, + worktree_path: str, + result: subprocess.CompletedProcess[str], +) -> CredentialRefreshError: + return CredentialRefreshError( + f"{backend_name} could not use its stored credential " + f"(matched {marker!r} in {where})", + operation=f"{backend_name} invocation", + phase="credential check", + command=cmd_str, + cwd=str(worktree_path), + stdout=result.stdout, + stderr=result.stderr, + exit_code=result.returncode, + suggestion=( + f"This is a credential failure, not a failed mutation: {backend_name} " + "reported that its stored login could not be used or refreshed. " + f"Re-authenticate with `helix sandbox login {backend}`, then resume " + "the run; nothing is wrong with the candidate's code." + ), + ) + + # --------------------------------------------------------------------------- # Rendered-mutation-prompt artifact # --------------------------------------------------------------------------- @@ -1685,6 +1825,30 @@ def invoke_claude_code( worktree_path=worktree_path, ) usage = _normalise_usage_stats(parsed) + # A backend can report an unusable credential and still exit 0 -- + # measured on codex-cli 0.130.0, whose refresh failure is swallowed + # entirely (exit 0, empty stderr, even at RUST_LOG=info). The + # envelope's own ``is_error`` flag is the only signal left on this + # path, so read it here rather than letting the failure pass as a + # successful-but-useless mutation. + evidence = _credential_failure_evidence(parsed, result) + if evidence is not None: + marker, where = evidence + logger.error( + "Credential failure detected for %s in %s: matched %r", + backend_name, + where, + marker, + ) + raise _credential_refresh_error( + backend=backend, + backend_name=backend_name, + marker=marker, + where=where, + cmd_str=cmd_str, + worktree_path=worktree_path, + result=result, + ) if backend == "claude": error_text = str(parsed.get("error", "")) if _looks_like_rate_limit(error_text): @@ -1707,6 +1871,30 @@ def invoke_claude_code( ) return parsed, usage + # Classify a credential failure ahead of the rate-limit and generic + # paths. The markers are disjoint from the rate-limit keywords, and + # "the login is unusable" is a strictly more actionable verdict than + # "the backend exited non-zero". + evidence = _credential_failure_evidence(parsed, result) + if evidence is not None: + marker, where = evidence + logger.error( + "Credential failure detected for %s (exit %d) in %s: matched %r", + backend_name, + result.returncode, + where, + marker, + ) + raise _credential_refresh_error( + backend=backend, + backend_name=backend_name, + marker=marker, + where=where, + cmd_str=cmd_str, + worktree_path=worktree_path, + result=result, + ) + rate_limit_source = result.stderr or result.stdout if _looks_like_rate_limit(rate_limit_source): logger.error( @@ -1868,6 +2056,16 @@ def mutate( except Exception: pass raise + except CredentialRefreshError: + # The stored login, not this candidate, is what failed. Clean up the + # orphaned worktree and re-raise so evolution.py can count it and name + # it as a credential failure instead of filing it under "the agent + # wrote bad code". + try: + remove_worktree(child) + except Exception: + pass + raise # NOTE: snapshot_candidate() is intentionally NOT called here. # The caller (evolution.py) is responsible for calling save_state() diff --git a/src/helix/sandbox.py b/src/helix/sandbox.py index 376aac9b..a30365ca 100644 --- a/src/helix/sandbox.py +++ b/src/helix/sandbox.py @@ -24,7 +24,11 @@ agent_state_env, agent_state_subdirs, ) -from helix.backends import BACKEND_AUTH_COMMANDS, DEFAULT_BACKEND_IMAGES +from helix.backends import ( + BACKEND_AUTH_COMMANDS, + DEFAULT_BACKEND_IMAGES, + backend_credential_warm_skip_reason, +) from helix.config import EvaluatorSidecarConfig, SandboxConfig @@ -1154,7 +1158,7 @@ def sandbox_auth_docker_args( agent_backend: str, *, image: str, - action: Literal["login", "status", "logout"], + action: Literal["login", "status", "logout", "warm"], network: str = "bridge", add_host_gateway: bool = False, extra_hosts: dict[str, str] | None = None, @@ -1199,7 +1203,7 @@ def sandbox_auth_docker_args( def run_sandbox_auth_command( agent_backend: str, *, - action: Literal["login", "status", "logout"], + action: Literal["login", "status", "logout", "warm"], image: str | None = None, network: str = "bridge", add_host_gateway: bool = False, @@ -1223,6 +1227,93 @@ def run_sandbox_auth_command( return subprocess.run(args, capture_output=True, text=True) +@dataclass(frozen=True) +class CredentialWarmResult: + """Outcome of one per-generation credential warm. + + ``warmed`` is True only when the warm container ran and exited cleanly. + ``skip_reason`` is set when the backend is deliberately not warmed; + ``detail`` carries the diagnosis when a warm was attempted and failed. + """ + + backend: str + warmed: bool + skip_reason: str | None = None + returncode: int | None = None + detail: str = "" + + @property + def skipped(self) -> bool: + return self.skip_reason is not None + + @property + def failed(self) -> bool: + return not self.warmed and self.skip_reason is None + + +#: Tail of the warm command's stderr kept for diagnosis. Backend CLIs report +#: refresh failures in prose, not by echoing the credential, but the cap keeps +#: an unexpectedly chatty CLI from pasting its whole state into the run log. +_WARM_DETAIL_CHARS = 400 + + +def warm_backend_credential( + agent_backend: str, *, sandbox: SandboxConfig +) -> CredentialWarmResult: + """Refresh *agent_backend*'s shared credential once, under a single writer. + + Runs the backend's registered ``warm`` command through the same sandboxed + auth container that ``helix sandbox status`` uses: one container, the + shared login volume mounted read-write, nothing else running against it. + Any refresh the CLI decides is due therefore happens exactly once and is + written back before candidates start, instead of N candidates racing to + spend the same single-use refresh token. + + The call is a no-op whenever the credential is already fresh -- the warm + command is chosen so the CLI does no work in that case (see + ``helix.backends``). Backends that need no warm return a skipped result + rather than starting a container. + + Never raises: a warm that cannot run is reported, not fatal. Candidates + may still succeed on the credential that is already there, so the run + continues either way and the caller decides how loudly to say so. + """ + skip_reason = backend_credential_warm_skip_reason(agent_backend) + if skip_reason is not None: + return CredentialWarmResult( + backend=agent_backend, warmed=False, skip_reason=skip_reason + ) + + try: + image = resolve_sandbox_image(sandbox, agent_backend) + result = run_sandbox_auth_command( + agent_backend, + action="warm", + image=image, + network=sandbox.network, + add_host_gateway=sandbox.add_host_gateway, + extra_hosts=sandbox.extra_hosts, + ) + except (OSError, ValueError, subprocess.SubprocessError) as exc: + return CredentialWarmResult( + backend=agent_backend, + warmed=False, + detail=f"{type(exc).__name__}: {exc}", + ) + + if result.returncode == 0: + return CredentialWarmResult( + backend=agent_backend, warmed=True, returncode=0 + ) + stderr = (result.stderr or "").strip() + return CredentialWarmResult( + backend=agent_backend, + warmed=False, + returncode=result.returncode, + detail=stderr[-_WARM_DETAIL_CHARS:], + ) + + def run_command( command: list[str], *, diff --git a/tests/integration/test_credential_warm_docker.py b/tests/integration/test_credential_warm_docker.py new file mode 100644 index 00000000..3b74cd84 --- /dev/null +++ b/tests/integration/test_credential_warm_docker.py @@ -0,0 +1,204 @@ +"""Container proof that the credential warm is real, and that it is free. + +The warm exists to move a due token refresh out of N racing candidates and into +one writer. Two claims have to hold against a real backend container or the +change is worse than useless: + +(a) it costs nothing -- no model call, no quota, on an operator's paid account + that would otherwise be charged once per generation; and +(b) it does nothing at all when the credential is already fresh. + +Both are asserted here by running the registered warm command with +``--network none``. A command that completes with no network cannot have +reached a model; a warm that leaves the volume byte-identical did no work. + +Credentials are synthetic and live in throwaway volumes (see ``conftest``); no +test logs in, and none can reach a real ``helix-auth-*`` volume. +""" + +from __future__ import annotations + +import subprocess + +import pytest + +from helix.backends import ( + BACKEND_AUTH_COMMANDS, + BACKENDS, + backend_credential_warm_skip_reason, +) + + +pytestmark = pytest.mark.docker_integration + + +CODEX_IMAGE = "ghcr.io/ke7/helix-evo-runner-codex:latest" + +# A well-shaped credential that no real service would ever accept. Only two +# things matter: that codex reads it as a stored ChatGPT login, and that its +# ``last_refresh`` is stamped now, so no refresh is due. The id_token is an +# unsigned JWT over invented claims -- codex parses it for the account fields +# and nothing here is a real token or can rotate a real grant. +SYNTHETIC_CODEX_ID_TOKEN = ( + "eyJhbGciOiJub25lIiwidHlwIjoiSldUIn0.eyJzdWIiOiJTWU5USEVUSUMiLCJleHAiOjQxMDI0NDQ4MDAsImVtYWlsIjoic3ludGhldGljQGV4YW1wbGUuaW52YWxpZCIsImh0dHBzOi8vYXBpLm9wZW5haS5jb20vYXV0aCI6eyJjaGF0Z3B0X3BsYW5fdHlwZSI6InBsdXMiLCJjaGF0Z3B0X2FjY291bnRfaWQiOiJTWU5USEVUSUMiLCJjaGF0Z3B0X3VzZXJfaWQiOiJTWU5USEVUSUMiLCJ1c2VyX2lkIjoiU1lOVEhFVElDIn19.SYNTHETIC" +) + +SYNTHETIC_FRESH_CODEX_AUTH = ( + "mkdir -p /home/node/.codex; " + "NOW=$(date -u +%Y-%m-%dT%H:%M:%S.000000Z); " + "printf '{\"OPENAI_API_KEY\":null,\"tokens\":{" + "\"id_token\":\"%s\",\"access_token\":\"SYNTHETIC\"," + "\"refresh_token\":\"SYNTHETIC\",\"account_id\":\"SYNTHETIC\"}," + "\"last_refresh\":\"%s\"}' " + f"\"{SYNTHETIC_CODEX_ID_TOKEN}\" \"$NOW\" " + "> /home/node/.codex/auth.json" +) + + +def _run_warm(*, image: str, volume: str, backend: str, timeout: int = 180): + """Run the backend's registered warm command the way ``helix.sandbox`` does. + + Same single-writer shape as ``sandbox_auth_docker_args``: one container, + the login volume read-write at ``/home/node``, nothing else attached. The + only deliberate difference is ``--network none``, which turns "this makes + no model call" from a claim into something the container can prove. + """ + args = [ + "docker", + "run", + "--rm", + "--network", + "none", + "--security-opt", + "no-new-privileges", + "--user", + "node", + "-v", + f"{volume}:/home/node:rw", + "-e", + "HOME=/home/node", + "-e", + "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", + image, + *BACKEND_AUTH_COMMANDS[backend]["warm"], + ] + return subprocess.run( + args, capture_output=True, text=True, check=False, timeout=timeout + ) + + +def _files(listing: set[str]) -> set[str]: + """Keep only regular files from a ``volume_listing`` result. + + Directories are filtered out deliberately: the codex CLI scaffolds a few + empty ones (``.codex/tmp``, ``.codex/memories``) the first time it starts + in a fresh volume, and every candidate would create the same ones on its + own. What must not move is the credential and the files beside it. + """ + return {line for line in listing if line.startswith("-")} + + +def _auth_digest(volume: str, image: str) -> str: + """Hash the stored credential without reading it. + + The digest is the whole point: it proves the file did or did not change + without anything ever printing, decoding, or logging its contents. + """ + result = subprocess.run( + [ + "docker", + "run", + "--rm", + "--network", + "none", + "-v", + f"{volume}:/home/node:ro", + "--user", + "node", + image, + "sh", + "-c", + "sha256sum /home/node/.codex/auth.json | cut -d' ' -f1", + ], + capture_output=True, + text=True, + check=True, + timeout=120, + ) + return result.stdout.strip() + + +@pytest.mark.timeout(300) +def test_codex_warm_is_free(require_image, throwaway_volume) -> None: + """No model call, no quota -- proved by completing with no network at all. + + This is the constraint that decides whether the warm may ship: it runs on + the operator's paid account once per generation, so a status/whoami command + that quietly cost a request would be worse than the race it prevents. A + container with no network namespace cannot reach any endpoint, so a clean + exit here is proof rather than assertion. + """ + image = require_image(CODEX_IMAGE) + volume = throwaway_volume(image, SYNTHETIC_FRESH_CODEX_AUTH) + + result = _run_warm(image=image, volume=volume, backend="codex") + + assert result.returncode == 0, result.stderr + + +@pytest.mark.timeout(300) +def test_codex_warm_is_a_no_op_on_a_fresh_credential( + require_image, throwaway_volume, volume_listing +) -> None: + """A credential that is not due for refresh must come back untouched. + + If the warm rewrote a fresh credential it would be doing the very thing it + exists to prevent -- spending a single-use refresh token nobody needed to + spend -- once per generation. + """ + image = require_image(CODEX_IMAGE) + volume = throwaway_volume(image, SYNTHETIC_FRESH_CODEX_AUTH) + before_files = _files(volume_listing(volume, image)) + before_digest = _auth_digest(volume, image) + + _run_warm(image=image, volume=volume, backend="codex") + + assert _auth_digest(volume, image) == before_digest + assert _files(volume_listing(volume, image)) == before_files + + +@pytest.mark.timeout(300) +def test_repeated_codex_warms_leave_no_residue( + require_image, throwaway_volume, volume_listing +) -> None: + """The warm runs every generation, so it must not accumulate. + + It also must not behave like a candidate: a command that opened a session + or wrote a rollout into the shared volume would hand the next candidate + state it did not create -- the contamination the per-candidate state work + on this branch removes. + """ + image = require_image(CODEX_IMAGE) + volume = throwaway_volume(image, SYNTHETIC_FRESH_CODEX_AUTH) + + _run_warm(image=image, volume=volume, backend="codex") + after_first = volume_listing(volume, image) + digest_first = _auth_digest(volume, image) + + for _ in range(2): + _run_warm(image=image, volume=volume, backend="codex") + + assert volume_listing(volume, image) == after_first + assert _auth_digest(volume, image) == digest_first + + +@pytest.mark.timeout(120) +def test_skipped_backends_have_no_warm_command_to_run() -> None: + """Skipping is a decision this suite is allowed to see. + + If a backend ever gains a warm command, it must gain container proof in + this file at the same time -- this assertion is what makes that fail loudly + instead of shipping an unmeasured per-generation call on a paid account. + """ + warmed = [b for b in BACKENDS if backend_credential_warm_skip_reason(b) is None] + assert warmed == ["codex"] diff --git a/tests/unit/test_credential_failure_classification.py b/tests/unit/test_credential_failure_classification.py new file mode 100644 index 00000000..24974b8f --- /dev/null +++ b/tests/unit/test_credential_failure_classification.py @@ -0,0 +1,274 @@ +"""A dead credential must not read as a candidate that wrote bad code. + +Before this, a run whose shared login stopped working looked exactly like a run +whose agent kept producing broken diffs: the mutation was abandoned, the slot +was dropped, and nothing said why. These tests pin the two halves of the fix +-- that the real wording the shipped CLIs emit is recognised, and that ordinary +agent output is not. + +Every positive string below is a phrase read out of the shipped binary of the +CLI it is attributed to, not invented for the test. The negatives are the +false-positive trap: candidate work that merely *mentions* tokens, auth, or a +401 must never be reported to an operator as a broken login. +""" + +from __future__ import annotations + +import json +import subprocess +from pathlib import Path +from typing import Any + +import pytest + +from helix.config import AgentConfig +from helix.exceptions import ( + CredentialRefreshError, + HelixError, + MutationError, + RateLimitError, +) +from helix.mutator import credential_failure_marker, invoke_claude_code + + +# Codex CLI 0.130.0 -- the four suffixes it appends to one prefix, plus the +# bare form. The "already used" variant is what losing a refresh race looks +# like from inside a candidate. +CODEX_ALREADY_USED = ( + "Your access token could not be refreshed because your refresh token was " + "already used. Please log out and sign in again." +) +CODEX_EXPIRED = ( + "Your access token could not be refreshed because your refresh token has " + "expired. Please log out and sign in again." +) +CODEX_REVOKED = ( + "Your access token could not be refreshed because your refresh token was " + "revoked. Please log out and sign in again." +) +CODEX_OTHER_ACCOUNT = ( + "Your access token could not be refreshed because you have since logged " + "out or signed in to another account. Please sign in again." +) +CODEX_BARE = ( + "Your access token could not be refreshed. Please log out and sign in again." +) +CODEX_GET_ACCOUNT = "failed to refresh token while getting account: 401" +CODEX_NO_ACCOUNT = ( + "ChatGPT account ID not available, please re-run `codex login`" +) +# OpenCode 1.14.24 -- thrown by the provider fetch wrapper. +OPENCODE_REFRESH_FAILED = "Token refresh failed: 400" +# Claude Code 2.1.138. +CLAUDE_OAUTH_REFRESH = "User OAuth refresh failed (HTTP 401): invalid_grant" +CLAUDE_INVALID_KEY = "API Error: 401 Invalid API key · Please run /login" + +CREDENTIAL_STRINGS = [ + CODEX_ALREADY_USED, + CODEX_EXPIRED, + CODEX_REVOKED, + CODEX_OTHER_ACCOUNT, + CODEX_BARE, + CODEX_GET_ACCOUNT, + CODEX_NO_ACCOUNT, + OPENCODE_REFRESH_FAILED, + CLAUDE_OAUTH_REFRESH, + CLAUDE_INVALID_KEY, +] + +# Things a candidate legitimately produces while working on code. None of them +# is a statement about HELIX's own login. +INNOCENT_STRINGS = [ + "", + "401", + "token", + "auth", + "refresh", + "TypeError: 'NoneType' object is not subscriptable", + "FAILED tests/test_auth.py::test_refresh_token_rotation - assert 0 == 1", + "+ def refresh_token(self) -> str:\n+ raise NotImplementedError", + "The test suite returned 401 for 3 requests; see auth.py line 88.", + "Added a refresh token cache so the client stops re-authenticating.", + "Error: 529 overloaded, please retry", + "You have exceeded your usage limit", +] + + +class TestMarkerRecognition: + @pytest.mark.parametrize("text", CREDENTIAL_STRINGS) + def test_real_cli_wording_is_recognised(self, text: str) -> None: + assert credential_failure_marker(text) is not None + + @pytest.mark.parametrize("text", CREDENTIAL_STRINGS) + def test_recognition_is_case_insensitive(self, text: str) -> None: + assert credential_failure_marker(text.upper()) is not None + + @pytest.mark.parametrize("text", INNOCENT_STRINGS) + def test_ordinary_candidate_output_is_not_a_credential_failure( + self, text: str + ) -> None: + assert credential_failure_marker(text) is None + + def test_marker_is_returned_as_evidence(self) -> None: + """The caller names what matched instead of asserting a bare verdict.""" + marker = credential_failure_marker(CODEX_ALREADY_USED) + assert marker == "your access token could not be refreshed" + + def test_embedded_in_a_larger_stream_is_still_found(self) -> None: + stream = "\n".join( + ["running 3 tests", CODEX_ALREADY_USED, "process exited"] + ) + assert credential_failure_marker(stream) is not None + + +class TestExceptionTaxonomy: + def test_credential_error_is_a_helix_error(self) -> None: + assert isinstance(CredentialRefreshError("x"), HelixError) + + def test_credential_error_is_not_a_mutation_error(self) -> None: + """``except MutationError`` in mutate() must not swallow it as a + failed mutation -- that is the exact conflation being removed.""" + assert not isinstance(CredentialRefreshError("x"), MutationError) + + def test_credential_error_is_not_a_rate_limit(self) -> None: + """A rate limit clears on its own; an unusable credential does not.""" + assert not isinstance(CredentialRefreshError("x"), RateLimitError) + + +def _patch_backend( + mocker: Any, *, returncode: int, stdout: str = "", stderr: str = "" +) -> None: + result = subprocess.CompletedProcess( + args=["backend"], returncode=returncode, stdout=stdout, stderr=stderr + ) + mocker.patch("helix.mutator.subprocess.run", return_value=result) + + +class TestInvocationClassification: + def test_non_zero_exit_with_cli_wording_on_stderr( + self, mocker: Any, tmp_path: Path + ) -> None: + _patch_backend(mocker, returncode=1, stderr=CODEX_ALREADY_USED) + with pytest.raises(CredentialRefreshError) as exc: + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + err = exc.value + assert err.exit_code == 1 + assert err.stderr == CODEX_ALREADY_USED + assert "credential" in err.suggestion.lower() + assert "helix sandbox login codex" in err.suggestion + + def test_zero_exit_is_error_envelope_is_read( + self, mocker: Any, tmp_path: Path + ) -> None: + """Codex swallows its own refresh failure: exit 0, empty stderr. + + Measured on codex-cli 0.130.0 against a synthetic credential whose + refresh was rejected -- the process exits 0 and prints nothing, even at + RUST_LOG=info. The envelope's ``is_error`` flag is the only signal + left, so it has to be read. + """ + stream = "\n".join( + [ + json.dumps({"type": "thread.started"}), + json.dumps({"type": "error", "is_error": True, + "message": CODEX_ALREADY_USED}), + ] + ) + _patch_backend(mocker, returncode=0, stdout=stream) + with pytest.raises(CredentialRefreshError) as exc: + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + assert exc.value.exit_code == 0 + assert "is_error" in str(exc.value) + + def test_claude_top_level_envelope_is_read( + self, mocker: Any, tmp_path: Path + ) -> None: + envelope = json.dumps( + { + "type": "result", + "subtype": "error_during_execution", + "is_error": True, + "result": CLAUDE_INVALID_KEY, + } + ) + _patch_backend(mocker, returncode=0, stdout=envelope) + with pytest.raises(CredentialRefreshError): + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="claude") + ) + + def test_is_error_alone_does_not_classify( + self, mocker: Any, tmp_path: Path + ) -> None: + """An ``is_error`` tool result is usually the agent's own failing + command. That is an ordinary code failure and must stay one.""" + stream = "\n".join( + [ + json.dumps( + { + "type": "tool_result", + "is_error": True, + "content": "pytest exited 1: 2 failed, 9 passed", + } + ), + json.dumps({"type": "turn.completed"}), + ] + ) + _patch_backend(mocker, returncode=0, stdout=stream) + parsed, _usage = invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + assert parsed["events"] + + def test_candidate_editing_oauth_code_is_not_classified( + self, mocker: Any, tmp_path: Path + ) -> None: + """A candidate whose own work is about refresh tokens must not be able + to talk HELIX into declaring the operator's login broken.""" + stream = "\n".join( + [ + json.dumps( + { + "type": "tool_result", + "is_error": True, + "content": ( + "FAILED tests/test_oauth.py::test_reuse - " + "expected the refresh token to be rejected" + ), + } + ) + ] + ) + _patch_backend(mocker, returncode=0, stdout=stream) + parsed, _usage = invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + assert parsed["events"] + + def test_ordinary_non_zero_exit_stays_a_mutation_error( + self, mocker: Any, tmp_path: Path + ) -> None: + _patch_backend( + mocker, + returncode=1, + stderr="Traceback (most recent call last): SyntaxError", + ) + with pytest.raises(MutationError): + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + + def test_rate_limit_still_wins_its_own_classification( + self, mocker: Any, tmp_path: Path + ) -> None: + _patch_backend( + mocker, returncode=1, stderr="Error: 529 overloaded please retry" + ) + with pytest.raises(RateLimitError): + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) diff --git a/tests/unit/test_credential_warm.py b/tests/unit/test_credential_warm.py new file mode 100644 index 00000000..9a61dec1 --- /dev/null +++ b/tests/unit/test_credential_warm.py @@ -0,0 +1,273 @@ +"""The per-generation credential warm: who gets one, and what it costs. + +Candidates share one login volume read-write. When a refresh comes due they +can each decide to refresh independently and spend the same single-use refresh +token; whoever loses that race can fail without saying anything. The warm +closes the window by doing the refresh once, in one container, before a +generation dispatches anything. + +Two properties are load-bearing and both are asserted here: the warm runs +through the existing single-writer auth path, and a backend is warmed only when +warming it is both useful and free. A backend with no warm command must carry +a written reason -- skipping is a claim about correctness, not an omission. +""" + +from __future__ import annotations + +import subprocess +from typing import Any + +import pytest + +from helix.backends import ( + BACKEND_AUTH_COMMANDS, + BACKENDS, + CREDENTIAL_WARM_SKIP_REASONS, + backend_credential_warm_skip_reason, +) +from helix.config import AgentConfig, EvaluatorConfig, HelixConfig, SandboxConfig +from helix.evolution import _warm_generation_credential +from helix.sandbox import ( + CredentialWarmResult, + sandbox_auth_docker_args, + warm_backend_credential, +) + + +WARMED_BACKENDS = ("codex",) +SKIPPED_BACKENDS = ("claude", "cursor", "gemini", "opencode") + + +def _completed(returncode: int, stderr: str = "") -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess( + args=["docker"], returncode=returncode, stdout="", stderr=stderr + ) + + +# --------------------------------------------------------------------------- +# Registry: every backend is either warmed or explained +# --------------------------------------------------------------------------- + + +class TestWarmRegistry: + def test_every_backend_is_warmed_or_has_a_reason(self) -> None: + """No backend may fall through silently. + + A missing warm command with no recorded reason is indistinguishable + from an oversight, which is exactly the state this change exists to + leave behind. + """ + for backend in BACKENDS: + has_warm = "warm" in BACKEND_AUTH_COMMANDS[backend] + has_reason = backend in CREDENTIAL_WARM_SKIP_REASONS + assert has_warm != has_reason, backend + + @pytest.mark.parametrize("backend", WARMED_BACKENDS) + def test_warmed_backend_has_no_skip_reason(self, backend: str) -> None: + assert backend_credential_warm_skip_reason(backend) is None + + @pytest.mark.parametrize("backend", SKIPPED_BACKENDS) + def test_skipped_backend_states_why(self, backend: str) -> None: + reason = backend_credential_warm_skip_reason(backend) + assert reason is not None + # A reason a maintainer cannot act on is not a reason. + assert len(reason) > 60 + + def test_unknown_backend_is_not_warmed(self) -> None: + assert backend_credential_warm_skip_reason("nope") is not None + + def test_codex_warm_is_not_login_status(self) -> None: + """``codex login status`` never takes the refresh path. + + Measured against codex-cli 0.130.0 with a synthetic credential in a + throwaway volume: it exits 0 and issues no request whether the stored + ``last_refresh`` is minutes or 30 days old. Warming with it would look + like protection while providing none, so the registry must not drift + back to it. + """ + warm = BACKEND_AUTH_COMMANDS["codex"]["warm"] + assert warm != BACKEND_AUTH_COMMANDS["codex"]["status"] + assert "login" not in " ".join(warm) + + +# --------------------------------------------------------------------------- +# The warm reuses the single-writer sandboxed auth path +# --------------------------------------------------------------------------- + + +class TestWarmUsesSingleWriterPath: + def test_warm_action_mounts_the_shared_login_volume_read_write(self) -> None: + args = sandbox_auth_docker_args( + "codex", image="img:latest", action="warm" + ) + assert "-v" in args + assert "helix-auth-codex:/home/node:rw" in args + # One container, one command, nothing else attached to the volume. + assert args[:3] == ["docker", "run", "--rm"] + + def test_warm_action_runs_the_registered_warm_command(self) -> None: + args = sandbox_auth_docker_args( + "codex", image="img:latest", action="warm" + ) + assert args[-len(BACKEND_AUTH_COMMANDS["codex"]["warm"]) :] == ( + BACKEND_AUTH_COMMANDS["codex"]["warm"] + ) + + def test_warm_action_is_rejected_for_a_backend_without_one(self) -> None: + with pytest.raises(ValueError): + sandbox_auth_docker_args("cursor", image="img:latest", action="warm") + + def test_warm_forwards_sandbox_network_settings( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + seen: dict[str, Any] = {} + + def _fake(backend: str, **kwargs: Any) -> subprocess.CompletedProcess[str]: + seen.update(kwargs) + seen["backend"] = backend + return _completed(0) + + monkeypatch.setattr("helix.sandbox.run_sandbox_auth_command", _fake) + sandbox = SandboxConfig( + enabled=True, + network="none", + add_host_gateway=True, + extra_hosts={"h": "1.2.3.4"}, + image="custom:tag", + ) + result = warm_backend_credential("codex", sandbox=sandbox) + + assert result.warmed is True + assert seen["backend"] == "codex" + assert seen["action"] == "warm" + assert seen["network"] == "none" + assert seen["add_host_gateway"] is True + assert seen["extra_hosts"] == {"h": "1.2.3.4"} + # A configured image must win, or the warm would refresh the credential + # with a different CLI build than the candidates use. + assert seen["image"] == "custom:tag" + + +# --------------------------------------------------------------------------- +# Failure of the warm is reported, never fatal +# --------------------------------------------------------------------------- + + +class TestWarmFailureIsNotFatal: + def test_skipped_backend_starts_no_container( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + def _boom(*_a: Any, **_k: Any) -> None: + raise AssertionError("a skipped backend must not run a container") + + monkeypatch.setattr("helix.sandbox.run_sandbox_auth_command", _boom) + result = warm_backend_credential( + "cursor", sandbox=SandboxConfig(enabled=True) + ) + assert result.skipped is True + assert result.warmed is False + assert result.failed is False + + def test_non_zero_exit_is_reported_not_raised( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + monkeypatch.setattr( + "helix.sandbox.run_sandbox_auth_command", + lambda *a, **k: _completed(3, "warm blew up"), + ) + result = warm_backend_credential( + "codex", sandbox=SandboxConfig(enabled=True) + ) + assert result.failed is True + assert result.returncode == 3 + assert "warm blew up" in result.detail + + def test_docker_exception_is_reported_not_raised( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + def _raise(*_a: Any, **_k: Any) -> None: + raise OSError("no docker here") + + monkeypatch.setattr("helix.sandbox.run_sandbox_auth_command", _raise) + result = warm_backend_credential( + "codex", sandbox=SandboxConfig(enabled=True) + ) + assert result.failed is True + assert "no docker here" in result.detail + + def test_detail_is_capped(self, monkeypatch: pytest.MonkeyPatch) -> None: + monkeypatch.setattr( + "helix.sandbox.run_sandbox_auth_command", + lambda *a, **k: _completed(1, "x" * 5000), + ) + result = warm_backend_credential( + "codex", sandbox=SandboxConfig(enabled=True) + ) + assert 0 < len(result.detail) <= 400 + + +# --------------------------------------------------------------------------- +# Once per generation, and only where there is a volume to warm +# --------------------------------------------------------------------------- + + +def _config(backend: str, *, sandboxed: bool) -> HelixConfig: + return HelixConfig( + objective="Improve the code", + evaluator=EvaluatorConfig(command="pytest -q"), + agent=AgentConfig(backend=backend), # type: ignore[arg-type] + sandbox=SandboxConfig(enabled=sandboxed), + ) + + +class TestGenerationWarm: + def test_unsandboxed_run_warms_nothing( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + """There is no HELIX-managed login volume to arbitrate without the sandbox.""" + + def _boom(*_a: Any, **_k: Any) -> None: + raise AssertionError("no warm without a sandbox") + + monkeypatch.setattr("helix.evolution.warm_backend_credential", _boom) + assert ( + _warm_generation_credential( + _config("codex", sandboxed=False), gen=1, announce_skip=True + ) + is None + ) + + def test_sandboxed_run_warms_the_configured_backend( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + calls: list[str] = [] + + def _fake(backend: str, **_k: Any) -> CredentialWarmResult: + calls.append(backend) + return CredentialWarmResult(backend=backend, warmed=True, returncode=0) + + monkeypatch.setattr("helix.evolution.warm_backend_credential", _fake) + result = _warm_generation_credential( + _config("codex", sandboxed=True), gen=4, announce_skip=False + ) + assert calls == ["codex"] + assert result is not None and result.warmed + + def test_failed_warm_returns_and_does_not_raise( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + monkeypatch.setattr( + "helix.evolution.warm_backend_credential", + lambda backend, **_k: CredentialWarmResult( + backend=backend, warmed=False, returncode=1, detail="boom" + ), + ) + result = _warm_generation_credential( + _config("codex", sandboxed=True), gen=2, announce_skip=False + ) + assert result is not None and result.failed + # The operator is told what protection was lost, not just that a + # command exited non-zero. + printed = capsys.readouterr().out + assert "refresh" in printed.lower() + assert "run continues" in printed.lower() diff --git a/tests/unit/test_credential_warm_loop.py b/tests/unit/test_credential_warm_loop.py new file mode 100644 index 00000000..516a6e18 --- /dev/null +++ b/tests/unit/test_credential_warm_loop.py @@ -0,0 +1,193 @@ +"""The warm's placement in the loop, and how a credential failure ends up +somewhere an operator will actually read it. + +Two things are asserted end-to-end against the real evolution loop: + +* the warm fires once per generation -- not once per run, because a run + outlives any refresh interval, and not once per candidate, because that is + the race it exists to prevent; and +* when a mutation dies on the shared credential, the run says so in its own + terms and keeps going, instead of filing the slot under "the agent wrote bad + code" and finishing silently. +""" + +from __future__ import annotations + +from typing import Any + +import pytest + +from helix.config import SandboxConfig +from helix.evolution import run_evolution +from helix.exceptions import CredentialRefreshError +from helix.sandbox import CredentialWarmResult +from tests.unit.test_evolution import ( # type: ignore[import-untyped] + all_mocks, # noqa: F401, F811 — re-exported pytest fixture + make_candidate, + make_config, + make_eval_result, +) + + +def _sandboxed(config: Any) -> Any: + return config.model_copy(update={"sandbox": SandboxConfig(enabled=True)}) + + +@pytest.fixture() +def warm_calls(mocker: Any) -> list[str]: + calls: list[str] = [] + + def _fake(backend: str, **_kwargs: Any) -> CredentialWarmResult: + calls.append(backend) + return CredentialWarmResult(backend=backend, warmed=True, returncode=0) + + mocker.patch("helix.evolution.warm_backend_credential", side_effect=_fake) + return calls + + +class TestWarmRunsOncePerGeneration: + def test_one_warm_per_generation( + self, mocker, tmp_path, all_mocks, warm_calls # noqa: F811 + ) -> None: + """Three generations, three warms. + + A single warm at startup would leave a long run unprotected the moment + the credential next goes stale mid-flight, so the count must track + generations rather than runs. + """ + seed = make_candidate("g0-s0") + all_mocks["create_seed_worktree"].return_value = seed + all_mocks["mutate"].return_value = None + all_mocks["run_evaluator"].side_effect = ( + lambda candidate, *a, **k: make_eval_result( + candidate.id, {"i1": 0.5, "i2": 0.5} + ) + ) + + config = _sandboxed( + make_config(max_generations=3, perfect_score_threshold=None) + ) + run_evolution(config, tmp_path, tmp_path / ".helix") + + assert warm_calls == ["claude", "claude", "claude"] + + def test_warm_precedes_every_mutation( + self, mocker, tmp_path, all_mocks # noqa: F811 + ) -> None: + """Ordering, not just counting: the credential is fresh before any + candidate could start refreshing it for itself.""" + order: list[str] = [] + mocker.patch( + "helix.evolution.warm_backend_credential", + side_effect=lambda backend, **_k: ( + order.append("warm"), + CredentialWarmResult(backend=backend, warmed=True, returncode=0), + )[1], + ) + + seed = make_candidate("g0-s0") + all_mocks["create_seed_worktree"].return_value = seed + all_mocks["mutate"].side_effect = lambda *a, **k: ( + order.append("mutate"), + None, + )[1] + all_mocks["run_evaluator"].side_effect = ( + lambda candidate, *a, **k: make_eval_result( + candidate.id, {"i1": 0.5, "i2": 0.5} + ) + ) + + config = _sandboxed( + make_config(max_generations=2, perfect_score_threshold=None) + ) + run_evolution(config, tmp_path, tmp_path / ".helix") + + assert order[0] == "warm" + assert order.count("warm") == 2 + for index, event in enumerate(order): + if event == "mutate": + assert "warm" in order[:index] + + def test_unsandboxed_run_warms_nothing( + self, mocker, tmp_path, all_mocks, warm_calls # noqa: F811 + ) -> None: + seed = make_candidate("g0-s0") + all_mocks["create_seed_worktree"].return_value = seed + all_mocks["mutate"].return_value = None + all_mocks["run_evaluator"].side_effect = ( + lambda candidate, *a, **k: make_eval_result( + candidate.id, {"i1": 0.5, "i2": 0.5} + ) + ) + + config = make_config(max_generations=2, perfect_score_threshold=None) + run_evolution(config, tmp_path, tmp_path / ".helix") + + assert warm_calls == [] + + +class TestCredentialFailureIsVisible: + def test_failure_is_named_and_the_run_survives( + self, + mocker, # noqa: F811 + tmp_path, + all_mocks, # noqa: F811 + warm_calls, + capsys: pytest.CaptureFixture[str], + ) -> None: + """A dead credential must not read as a run that merely found nothing. + + The per-slot error names the cause, and the permanent end-of-run + summary repeats it after the live display is gone -- an operator who + looks only at the tail of the run still learns that the login, not the + code, is what failed. + """ + seed = make_candidate("g0-s0") + all_mocks["create_seed_worktree"].return_value = seed + all_mocks["run_evaluator"].side_effect = ( + lambda candidate, *a, **k: make_eval_result( + candidate.id, {"i1": 0.5, "i2": 0.5} + ) + ) + all_mocks["mutate"].side_effect = CredentialRefreshError( + "Codex CLI could not use its stored credential " + "(matched 'your access token could not be refreshed' in stderr)", + suggestion="Re-authenticate with `helix sandbox login codex`.", + ) + + config = _sandboxed( + make_config(max_generations=2, perfect_score_threshold=None) + ) + # The run completes rather than crashing: candidates may still work. + result = run_evolution(config, tmp_path, tmp_path / ".helix") + assert result.best_candidate.id == "g0-s0" + + # Rich wraps the console output; compare on collapsed whitespace. + out = " ".join(capsys.readouterr().out.lower().split()) + assert "credential" in out + assert "not a failure of the candidate's code" in out + assert "helix sandbox login" in out + + def test_clean_run_says_nothing_about_credentials( + self, + mocker, # noqa: F811 + tmp_path, + all_mocks, # noqa: F811 + warm_calls, + capsys: pytest.CaptureFixture[str], + ) -> None: + seed = make_candidate("g0-s0") + all_mocks["create_seed_worktree"].return_value = seed + all_mocks["mutate"].return_value = None + all_mocks["run_evaluator"].side_effect = ( + lambda candidate, *a, **k: make_eval_result( + candidate.id, {"i1": 0.5, "i2": 0.5} + ) + ) + + config = _sandboxed( + make_config(max_generations=2, perfect_score_threshold=None) + ) + run_evolution(config, tmp_path, tmp_path / ".helix") + + assert "credential" not in capsys.readouterr().out.lower() From 18a19fbc5301c0177a5971ab7a67f014c7af50c5 Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Wed, 9 Sep 2026 17:00:56 -0700 Subject: [PATCH 03/16] fix(merge): route a credential failure through the merge gate the way mutate() does CredentialRefreshError is a HelixError, not a MutationError, so merge()'s existing clauses let it escape with the merge worktree still on disk, and neither the merge call site in run_evolution nor the CLI caught it: a stale login that surfaced at the merge gate killed the run with a raw traceback and no end-of-run credential summary. - merger.merge(): new clause labels the operation, removes the child worktree and re-raises, mirroring mutate(). - evolution.run_evolution(): the merge call site catches it, records the slot in the run's CredentialFailureLog, prints the panel, and falls through to mutation so the generation continues. - cli.evolve / cli.resume: last-resort handler renders the panel plus a re-login/resume hint and exits 2 instead of dumping a traceback. Tests cover the merge() cleanup+re-raise, the end-to-end merge path naming the failure in the summary, and both CLI commands. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4 --- src/helix/cli.py | 28 ++++++++++- src/helix/evolution.py | 57 ++++++++++++++++------- src/helix/merger.py | 19 +++++++- tests/unit/test_cli_credential_failure.py | 51 ++++++++++++++++++++ tests/unit/test_credential_warm_loop.py | 54 +++++++++++++++++++++ tests/unit/test_merger.py | 32 +++++++++++++ 6 files changed, 223 insertions(+), 18 deletions(-) create mode 100644 tests/unit/test_cli_credential_failure.py diff --git a/src/helix/cli.py b/src/helix/cli.py index 539a28bf..c0c38124 100644 --- a/src/helix/cli.py +++ b/src/helix/cli.py @@ -26,7 +26,12 @@ print_warning, render_frontier_table, ) -from helix.exceptions import RateLimitError, ResumeIncompatibleError, print_helix_error +from helix.exceptions import ( + CredentialRefreshError, + RateLimitError, + ResumeIncompatibleError, + print_helix_error, +) from helix.lineage import load_lineage from helix.population import EvalResult, FrontierType, ParetoFrontier, Candidate from helix.state import load_state, save_state @@ -694,6 +699,18 @@ def evolve( "Run [cyan]helix resume[/cyan] to continue when rate limits clear." ) raise SystemExit(2) + except CredentialRefreshError as exc: + # Every in-loop path handles this itself (the slot is skipped and the + # run continues), so reaching here means a path that does not. Show + # the panel with its suggestion instead of a raw traceback. + logger.error("Credential failure escaped the evolution loop: %s", exc) + print_helix_error(exc) + print_error( + "Evolution state has been saved. Re-authenticate with " + f"[cyan]helix sandbox login {config.agent.backend}[/cyan] if the " + "login is stale, then run [cyan]helix resume[/cyan]." + ) + raise SystemExit(2) except KeyboardInterrupt: _handle_keyboard_interrupt(project_root) else: @@ -1236,6 +1253,15 @@ def resume(config_path: str, project_dir: Path | None) -> None: "Run [cyan]helix resume[/cyan] again when rate limits clear." ) raise SystemExit(2) + except CredentialRefreshError as exc: + logger.error("Credential failure escaped the resumed loop: %s", exc) + print_helix_error(exc) + print_error( + "Evolution state has been saved. Re-authenticate with " + f"[cyan]helix sandbox login {config.agent.backend}[/cyan] if the " + "login is stale, then run [cyan]helix resume[/cyan] again." + ) + raise SystemExit(2) except KeyboardInterrupt: _handle_keyboard_interrupt(project_root) else: diff --git a/src/helix/evolution.py b/src/helix/evolution.py index 44d7e03c..1eb13413 100644 --- a/src/helix/evolution.py +++ b/src/helix/evolution.py @@ -2508,22 +2508,47 @@ def _has_val_support_overlap(i: str, j: str) -> bool: f"diff form for this merge." ) - merged = merge( - candidate_a=a, - candidate_b=b, - new_id=merge_id, - config=config, - base_dir=worktrees_dir, - background=config.agent.background, - eval_result_a=era, - eval_result_b=erb, - prepare_worktree=lambda cand: ( - _refresh_and_snapshot_protected_evaluator_files( - cand, config, project_root - ) - ), - ancestor=ancestor_candidate, - ) + try: + merged = merge( + candidate_a=a, + candidate_b=b, + new_id=merge_id, + config=config, + base_dir=worktrees_dir, + background=config.agent.background, + eval_result_a=era, + eval_result_b=erb, + prepare_worktree=lambda cand: ( + _refresh_and_snapshot_protected_evaluator_files( + cand, config, project_root + ) + ), + ancestor=ancestor_candidate, + ) + except CredentialRefreshError as _merge_cred_exc: + # Same treatment the proposal worker gives a + # mutation: the merge worktree is already cleaned + # up by merge(); count and name the failure, then + # fall through to mutation so the run continues. + merged = None + credential_failures.record( + merge_id, str(_merge_cred_exc) + ) + print_helix_error(_merge_cred_exc) + logger.error( + "Merge %s (%s + %s, gen %d) failed on the shared " + "%s credential, not on its code: %s", + merge_id, a.id, b.id, gen, + backend_display_name(config.agent.backend), + _merge_cred_exc, + ) + print_error( + f"Merge [bold]{merge_id}[/bold] failed because the " + f"shared {backend_display_name(config.agent.backend)} " + f"credential could not be used or refreshed — this " + f"is a login failure, not a failure of the merged " + f"code. Falling through to mutation." + ) if merged is None: # GEPA parity (M2/B3): merge operator failed before diff --git a/src/helix/merger.py b/src/helix/merger.py index 2d7847b2..f4e048db 100644 --- a/src/helix/merger.py +++ b/src/helix/merger.py @@ -10,7 +10,12 @@ from helix.population import Candidate, EvalResult from helix.config import HelixConfig from helix.worktree import clone_candidate, snapshot_candidate, remove_worktree, get_diff # noqa: F401 -from helix.exceptions import MutationError, RateLimitError, print_helix_error +from helix.exceptions import ( + CredentialRefreshError, + MutationError, + RateLimitError, + print_helix_error, +) from helix.mutator import invoke_claude_code, AUTONOMOUS_SYSTEM_PROMPT, _turn_budget_section # --------------------------------------------------------------------------- @@ -361,6 +366,18 @@ def merge( except Exception: pass raise + except CredentialRefreshError as exc: + # The stored login, not this merge, is what failed. Mirror + # ``mutate()``: clean up the orphaned worktree and re-raise so the + # merge call site in evolution.py can count it as a credential + # failure and fall through to mutation instead of dying with a + # traceback and a leaked worktree. + exc.operation = f"merge {new_id} ({candidate_a.id} + {candidate_b.id})" + try: + remove_worktree(child) + except Exception: + pass + raise # NOTE: snapshot_candidate() is intentionally NOT called here. # The caller (evolution.py) is responsible for calling save_state() diff --git a/tests/unit/test_cli_credential_failure.py b/tests/unit/test_cli_credential_failure.py new file mode 100644 index 00000000..7da39d85 --- /dev/null +++ b/tests/unit/test_cli_credential_failure.py @@ -0,0 +1,51 @@ +"""``helix evolve`` / ``helix resume`` never let a credential failure escape +as a raw traceback. + +Every in-loop path handles :class:`CredentialRefreshError` itself, so these +handlers are the last line: if a future path forgets, the operator still gets +the error panel with its suggestion and a resume hint, not a stack dump. +""" + +from __future__ import annotations + +from pathlib import Path + +import pytest +from click.testing import CliRunner + +from helix.cli import cli +from helix.exceptions import CredentialRefreshError + + +def _make_project(tmp_path: Path) -> Path: + (tmp_path / "helix.toml").write_text( + 'objective = "test"\n\n[evaluator]\ncommand = "true"\n' + ) + return tmp_path + + +@pytest.mark.parametrize("command", ["evolve", "resume"]) +def test_escaped_credential_error_is_a_panel_not_a_traceback( + mocker, tmp_path: Path, command: str +) -> None: + project = _make_project(tmp_path) + # Both commands import run_evolution lazily inside the function body. + mocker.patch( + "helix.evolution.run_evolution", + side_effect=CredentialRefreshError( + "Codex CLI could not use its stored credential", + suggestion="Re-authenticate with `helix sandbox login codex`.", + ), + ) + if command == "resume": + mocker.patch("helix.cli.load_state", return_value=None) + + result = CliRunner().invoke(cli, [command, "--dir", str(project)]) + + assert result.exit_code == 2, result.output + assert result.exception is None or isinstance(result.exception, SystemExit) + assert "Traceback" not in result.output + out = " ".join(result.output.lower().split()) + assert "credential" in out + assert "helix sandbox login" in out + assert "helix resume" in out diff --git a/tests/unit/test_credential_warm_loop.py b/tests/unit/test_credential_warm_loop.py index 516a6e18..5509c12c 100644 --- a/tests/unit/test_credential_warm_loop.py +++ b/tests/unit/test_credential_warm_loop.py @@ -168,6 +168,60 @@ def test_failure_is_named_and_the_run_survives( assert "not a failure of the candidate's code" in out assert "helix sandbox login" in out + def test_merge_credential_failure_is_named_and_the_run_survives( + self, + mocker, # noqa: F811 + tmp_path, + all_mocks, # noqa: F811 + warm_calls, + capsys: pytest.CaptureFixture[str], + ) -> None: + """The merge gate fires before any mutation in a generation, so a + stale login can surface there first. It must get the same treatment + as a mutation: counted, named, and the run continues into mutation + rather than dying with a traceback.""" + seed = make_candidate("g0-s0") + child = make_candidate("g1-s1", generation=1) + all_mocks["create_seed_worktree"].return_value = seed + all_mocks["mutate"].return_value = child + all_mocks["merge"].side_effect = CredentialRefreshError( + "Codex CLI could not use its stored credential " + "(matched 'your access token could not be refreshed' in stderr)", + suggestion="Re-authenticate with `helix sandbox login codex`.", + ) + all_mocks["find_merge_triplet"].return_value = ("g0-s0", "g1-s1", "g0-s0") + + def run_eval(candidate, config, split=None, instances=None, **kwargs): + if candidate.id == "g1-s1": + return make_eval_result("g1-s1", {"i1": 0.9, "i2": 0.5}) + return make_eval_result(candidate.id, {"i1": 0.5, "i2": 0.8}) + + all_mocks["run_evaluator"].side_effect = run_eval + + config = _sandboxed( + make_config( + max_generations=2, + merge_enabled=True, + max_merge_invocations=5, + merge_val_overlap_floor=1, + max_evaluations=10000, + ) + ) + result = run_evolution(config, tmp_path, tmp_path / ".helix") + + all_mocks["merge"].assert_called_once() + # The run went on to mutate after the merge died on the credential. + assert all_mocks["mutate"].call_count >= 1 + assert result.best_candidate is not None + + out = " ".join(capsys.readouterr().out.lower().split()) + assert "merge" in out + assert "credential" in out + assert "not a failure of the merged code" in out + # The end-of-run summary names the merge slot, not just the mutation. + assert "1 mutation(s) failed on the shared" in out + assert "helix sandbox login" in out + def test_clean_run_says_nothing_about_credentials( self, mocker, # noqa: F811 diff --git a/tests/unit/test_merger.py b/tests/unit/test_merger.py index 49156291..7f891bc2 100644 --- a/tests/unit/test_merger.py +++ b/tests/unit/test_merger.py @@ -5,8 +5,11 @@ import random from pathlib import Path +import pytest + from helix.population import Candidate, EvalResult from helix.config import HelixConfig, EvaluatorConfig +from helix.exceptions import CredentialRefreshError from helix.mutator import MutationError from helix.merger import ( build_merge_prompt, @@ -305,6 +308,35 @@ def test_removes_worktree_on_failure(self, mocker): mock_remove.assert_called_once_with(child) + def test_credential_error_removes_worktree_and_reraises(self, mocker): + """A dead login must not leak the merge worktree or become a traceback. + + ``CredentialRefreshError`` is deliberately not a ``MutationError``, so + without its own clause it escaped ``merge()`` with the child worktree + still on disk. Mirror ``mutate()``: clean up, label the operation, + re-raise so evolution.py can count it as a credential failure. + """ + ca = make_candidate("g0-s0") + cb = make_candidate("g0-s1") + config = make_config() + + child = make_candidate("g1-m0") + mocker.patch("helix.merger.clone_candidate", return_value=child) + mocker.patch("helix.merger.get_diff", return_value="some diff") + mocker.patch( + "helix.merger.invoke_claude_code", + side_effect=CredentialRefreshError("login is dead"), + ) + mock_remove = mocker.patch("helix.merger.remove_worktree") + mock_snapshot = mocker.patch("helix.merger.snapshot_candidate") + + with pytest.raises(CredentialRefreshError) as exc: + merge(ca, cb, "g1-m0", config, Path("/tmp")) + + assert exc.value.operation == "merge g1-m0 (g0-s0 + g0-s1)" + mock_remove.assert_called_once_with(child) + mock_snapshot.assert_not_called() + def test_snapshot_not_called_by_merge_on_success(self, mocker): """merge() must NOT call snapshot_candidate — the caller owns that step. From c69ff7cf7d71c12e0de469cffb48e3d9fdc9b2f0 Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Wed, 9 Sep 2026 17:02:52 -0700 Subject: [PATCH 04/16] fix(sandbox): bound the per-generation credential warm and stop its container on timeout The warm ran on the main thread through run_sandbox_auth_command, whose non-interactive path was a bare subprocess.run with no timeout and a docker run with no --name. A token-refresh exchange that blackholed therefore blocked the whole evolution loop before any candidate was dispatched, and Ctrl-C left the unnamed container running against the shared login volume. - run_sandbox_auth_command: accepts timeout and container_name; on TimeoutExpired the named container is force-removed before re-raising, since killing the docker client does not stop the container. - sandbox_auth_docker_args: emits --name when a container name is given. - warm_backend_credential: always runs under credential_warm_timeout() (a 300s cap that sandbox.timeout_seconds can only tighten), names its container helix-warm--, and reports a timeout as a distinct, non-fatal CredentialWarmResult(timed_out=True). - _warm_generation_credential: says "timed out" instead of "(exit None)". Tests mock subprocess.run raising TimeoutExpired and assert the result is reported (not raised), the timeout applied, and the container removed. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4 --- src/helix/evolution.py | 8 ++- src/helix/sandbox.py | 59 +++++++++++++++++- tests/unit/test_credential_warm.py | 98 ++++++++++++++++++++++++++++++ 3 files changed, 163 insertions(+), 2 deletions(-) diff --git a/src/helix/evolution.py b/src/helix/evolution.py index 1eb13413..8747702c 100644 --- a/src/helix/evolution.py +++ b/src/helix/evolution.py @@ -1527,9 +1527,15 @@ def _warm_generation_credential( return result detail = f" Detail: {result.detail}" if result.detail else "" + if result.timed_out: + cause = "timed out" + elif result.returncode is not None: + cause = f"exit {result.returncode}" + else: + cause = "could not start" message = ( f"Credential warm for {display} did not complete before generation " - f"{gen} (exit {result.returncode}). Candidates in this generation will " + f"{gen} ({cause}). Candidates in this generation will " "each decide for themselves whether to refresh the shared login, and " "if a refresh is due they can spend the same single-use refresh token " "at once -- the losers of that race can fail without reporting an " diff --git a/src/helix/sandbox.py b/src/helix/sandbox.py index a30365ca..9ae22790 100644 --- a/src/helix/sandbox.py +++ b/src/helix/sandbox.py @@ -1163,6 +1163,7 @@ def sandbox_auth_docker_args( add_host_gateway: bool = False, extra_hosts: dict[str, str] | None = None, interactive: bool = False, + container_name: str | None = None, ) -> list[str]: try: command = BACKEND_AUTH_COMMANDS[agent_backend][action] @@ -1193,6 +1194,8 @@ def sandbox_auth_docker_args( "-e", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", ] + if container_name: + args.extend(["--name", container_name]) if interactive: args.insert(2, "-it") args.append(image) @@ -1209,7 +1212,18 @@ def run_sandbox_auth_command( add_host_gateway: bool = False, extra_hosts: dict[str, str] | None = None, interactive: bool = False, + timeout: float | None = None, + container_name: str | None = None, ) -> subprocess.CompletedProcess[str]: + """Run one auth-related command against the shared login volume. + + *timeout* applies to the non-interactive path only and is enforced on the + ``docker run`` client. Killing the client does not stop the container, so + when a *container_name* is given the container is force-removed on + timeout before :class:`subprocess.TimeoutExpired` propagates; without a + name there is nothing to address and the container is left to exit on its + own. + """ docker_image = image or resolve_sandbox_image( SandboxConfig(enabled=True), agent_backend ) @@ -1221,10 +1235,18 @@ def run_sandbox_auth_command( add_host_gateway=add_host_gateway, extra_hosts=extra_hosts, interactive=interactive, + container_name=container_name, ) if interactive: return subprocess.run(args, text=True) - return subprocess.run(args, capture_output=True, text=True) + try: + return subprocess.run( + args, capture_output=True, text=True, timeout=timeout + ) + except subprocess.TimeoutExpired: + if container_name: + _run_docker(["docker", "rm", "-f", container_name], check=False) + raise @dataclass(frozen=True) @@ -1241,6 +1263,7 @@ class CredentialWarmResult: skip_reason: str | None = None returncode: int | None = None detail: str = "" + timed_out: bool = False @property def skipped(self) -> bool: @@ -1256,6 +1279,21 @@ def failed(self) -> bool: #: an unexpectedly chatty CLI from pasting its whole state into the run log. _WARM_DETAIL_CHARS = 400 +#: Upper bound on one credential warm, in seconds. The warm is a single +#: token-refresh exchange plus, on first use, an image pull; it is never the +#: long-running agent turn that ``sandbox.timeout_seconds`` is sized for. A +#: stalled refresh must not wedge the whole evolution loop before any candidate +#: is dispatched, so the bound always applies -- ``sandbox.timeout_seconds`` +#: can only tighten it. +CREDENTIAL_WARM_TIMEOUT_SECONDS = 300 + + +def credential_warm_timeout(sandbox: SandboxConfig) -> float: + """Return the timeout for one warm: the fixed cap, tightened by the sandbox's.""" + if sandbox.timeout_seconds is not None: + return float(min(sandbox.timeout_seconds, CREDENTIAL_WARM_TIMEOUT_SECONDS)) + return float(CREDENTIAL_WARM_TIMEOUT_SECONDS) + def warm_backend_credential( agent_backend: str, *, sandbox: SandboxConfig @@ -1284,6 +1322,10 @@ def warm_backend_credential( backend=agent_backend, warmed=False, skip_reason=skip_reason ) + timeout = credential_warm_timeout(sandbox) + # Named so a timed-out warm can be stopped rather than left running + # against the shared login volume after the client has given up on it. + container_name = f"helix-warm-{agent_backend}-{uuid.uuid4().hex[:12]}" try: image = resolve_sandbox_image(sandbox, agent_backend) result = run_sandbox_auth_command( @@ -1293,6 +1335,21 @@ def warm_backend_credential( network=sandbox.network, add_host_gateway=sandbox.add_host_gateway, extra_hosts=sandbox.extra_hosts, + timeout=timeout, + container_name=container_name, + ) + except subprocess.TimeoutExpired: + # Non-fatal, like every other warm failure: the candidates fall back + # to refreshing for themselves. Reported distinctly because "the + # refresh hung" points somewhere different from "the refresh failed". + return CredentialWarmResult( + backend=agent_backend, + warmed=False, + timed_out=True, + detail=( + f"warm did not finish within {timeout:.0f}s; the warm " + "container was stopped" + ), ) except (OSError, ValueError, subprocess.SubprocessError) as exc: return CredentialWarmResult( diff --git a/tests/unit/test_credential_warm.py b/tests/unit/test_credential_warm.py index 9a61dec1..3d550c0f 100644 --- a/tests/unit/test_credential_warm.py +++ b/tests/unit/test_credential_warm.py @@ -28,7 +28,9 @@ from helix.config import AgentConfig, EvaluatorConfig, HelixConfig, SandboxConfig from helix.evolution import _warm_generation_credential from helix.sandbox import ( + CREDENTIAL_WARM_TIMEOUT_SECONDS, CredentialWarmResult, + credential_warm_timeout, sandbox_auth_docker_args, warm_backend_credential, ) @@ -206,6 +208,102 @@ def test_detail_is_capped(self, monkeypatch: pytest.MonkeyPatch) -> None: assert 0 < len(result.detail) <= 400 +# --------------------------------------------------------------------------- +# A stalled warm cannot wedge the loop +# --------------------------------------------------------------------------- + + +class TestWarmIsBounded: + """The warm runs on the main thread before any candidate is dispatched. + + Without a timeout, a token-refresh HTTP call that blackholes blocks the + whole evolution loop indefinitely with nothing but a debug log line, and + the unnamed container keeps running against the shared login volume + after the operator gives up and hits Ctrl-C. + """ + + def test_timeout_always_applies(self) -> None: + assert credential_warm_timeout(SandboxConfig(enabled=True)) == ( + CREDENTIAL_WARM_TIMEOUT_SECONDS + ) + + def test_sandbox_timeout_can_only_tighten_the_cap(self) -> None: + assert credential_warm_timeout( + SandboxConfig(enabled=True, timeout_seconds=30) + ) == 30.0 + assert credential_warm_timeout( + SandboxConfig(enabled=True, timeout_seconds=10_000) + ) == CREDENTIAL_WARM_TIMEOUT_SECONDS + + def test_warm_runs_with_a_timeout_and_a_named_container( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + seen: dict[str, Any] = {} + + def _fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: + seen["args"] = args + seen.update(kwargs) + return _completed(0) + + monkeypatch.setattr("helix.sandbox.subprocess.run", _fake_run) + result = warm_backend_credential( + "codex", sandbox=SandboxConfig(enabled=True, timeout_seconds=45) + ) + + assert result.warmed is True + assert seen["timeout"] == 45.0 + name = seen["args"][seen["args"].index("--name") + 1] + assert name.startswith("helix-warm-codex-") + + def test_timeout_is_reported_not_raised_and_the_container_is_stopped( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + removed: list[list[str]] = [] + + def _hang(args: list[str], **kwargs: Any) -> None: + raise subprocess.TimeoutExpired(cmd=args, timeout=kwargs["timeout"]) + + def _fake_docker(args: list[str], **_k: Any) -> subprocess.CompletedProcess[str]: + removed.append(args) + return _completed(0) + + monkeypatch.setattr("helix.sandbox.subprocess.run", _hang) + monkeypatch.setattr("helix.sandbox._run_docker", _fake_docker) + + result = warm_backend_credential( + "codex", sandbox=SandboxConfig(enabled=True, timeout_seconds=7) + ) + + assert result.failed is True + assert result.timed_out is True + assert result.warmed is False + assert result.skipped is False + assert "7s" in result.detail + # Killing the docker client does not stop the container; the named + # container must be force-removed so it stops touching the volume. + assert len(removed) == 1 + assert removed[0][:3] == ["docker", "rm", "-f"] + assert removed[0][3].startswith("helix-warm-codex-") + + def test_timed_out_warm_is_a_warning_in_the_loop( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + monkeypatch.setattr( + "helix.evolution.warm_backend_credential", + lambda backend, **_k: CredentialWarmResult( + backend=backend, warmed=False, timed_out=True, detail="slow" + ), + ) + result = _warm_generation_credential( + _config("codex", sandboxed=True), gen=3, announce_skip=False + ) + assert result is not None and result.failed and result.timed_out + printed = " ".join(capsys.readouterr().out.lower().split()) + assert "timed out" in printed + assert "exit none" not in printed + assert "run continues" in printed + + # --------------------------------------------------------------------------- # Once per generation, and only where there is a volume to warm # --------------------------------------------------------------------------- From 0f66bc709e28c64e8ce7adf70455f2a4a5dd6692 Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Wed, 9 Sep 2026 17:06:17 -0700 Subject: [PATCH 05/16] fix(mutator): retry once on a lost refresh race instead of reporting a dead login Codex's "...because your refresh token was already used" is the exact outcome of losing the refresh race this PR exists to handle: another candidate has just stored a refreshed credential in the shared volume. It was classified identically to an expired or revoked credential -- slot dropped, worktree deleted, no retry, and the operator told to re-run `helix sandbox login codex` over a login that was fine. - mutator: the already-used suffix is a separate, transient marker matched ahead of the generic prefix; CredentialRefreshError carries `transient`. invoke_claude_code() wraps the run+classify step in one _attempt() and, on a transient failure, retries exactly once against the now-refreshed credential. A second loss is raised with a suggestion that points at `helix resume` and does not instruct a re-login. - evolution: CredentialFailureLog records transience; the per-slot message and the end-of-run summary say "lost a refresh race ... run helix resume" when every failure was transient, and keep the re-login advice otherwise. Tests cover the marker split, retry-then-success (exit 1 and the exit-0 envelope path), retry-then-failure without a re-login instruction, that expired/revoked/bare wording is never retried, and the loop-level summary. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4 --- src/helix/evolution.py | 68 +++- src/helix/exceptions.py | 8 + src/helix/mutator.py | 374 +++++++++++------- .../test_credential_failure_classification.py | 135 ++++++- tests/unit/test_credential_warm_loop.py | 38 ++ 5 files changed, 449 insertions(+), 174 deletions(-) diff --git a/src/helix/evolution.py b/src/helix/evolution.py index 8747702c..6fcc0d10 100644 --- a/src/helix/evolution.py +++ b/src/helix/evolution.py @@ -1462,12 +1462,14 @@ class CredentialFailureLog: per-slot error that has already scrolled past by the time the run ends. """ - entries: list[tuple[str, str]] = field(default_factory=list) + entries: list[tuple[str, str, bool]] = field(default_factory=list) _lock: threading.Lock = field(default_factory=threading.Lock, repr=False) - def record(self, candidate_id: str, message: str) -> None: + def record( + self, candidate_id: str, message: str, *, transient: bool = False + ) -> None: with self._lock: - self.entries.append((candidate_id, message)) + self.entries.append((candidate_id, message, transient)) def __len__(self) -> int: with self._lock: @@ -1475,12 +1477,17 @@ def __len__(self) -> int: def candidate_ids(self) -> list[str]: with self._lock: - return [candidate_id for candidate_id, _ in self.entries] + return [candidate_id for candidate_id, _, _ in self.entries] def last_message(self) -> str: with self._lock: return self.entries[-1][1] if self.entries else "" + def all_transient(self) -> bool: + """True when every failure was a lost refresh race, not a dead login.""" + with self._lock: + return bool(self.entries) and all(t for _, _, t in self.entries) + def _warm_generation_credential( config: HelixConfig, *, gen: int, announce_skip: bool @@ -1685,7 +1692,9 @@ def _run_proposal_worker( # Name the failure for what it is. Without this the slot is # indistinguishable from a mutation that produced bad code, # and a whole generation can die quietly on a broken login. - credential_failures.record(_new_id, str(_mu_exc)) + credential_failures.record( + _new_id, str(_mu_exc), transient=_mu_exc.transient + ) logger.error( "Mutation %s (parent: %s, gen %d) failed on the shared " "%s credential, not on its code: %s", @@ -1695,8 +1704,14 @@ def _run_proposal_worker( print_error( f"Mutation [bold]{_new_id}[/bold] failed because the shared " f"{backend_display_name(config.agent.backend)} credential " - f"could not be used or refreshed — this is a login failure, " - f"not a failure of the candidate's code." + + ( + "was refreshed by another candidate first and the " + "retry also failed" + if _mu_exc.transient + else "could not be used or refreshed" + ) + + " — this is a login failure, not a failure of the " + "candidate's code." ) else: print_error( @@ -2538,7 +2553,9 @@ def _has_val_support_overlap(i: str, j: str) -> bool: # fall through to mutation so the run continues. merged = None credential_failures.record( - merge_id, str(_merge_cred_exc) + merge_id, + str(_merge_cred_exc), + transient=_merge_cred_exc.transient, ) print_helix_error(_merge_cred_exc) logger.error( @@ -3607,15 +3624,32 @@ def _drop_duplicate_child(gated: GatedProposal) -> bool: # that outlives the live display. if credential_failures: _failed_ids = ", ".join(credential_failures.candidate_ids()) - print_error( - f"{len(credential_failures)} mutation(s) failed on the shared " - f"{backend_display_name(config.agent.backend)} credential, not on " - f"their code: {_failed_ids}. The backend reported that its stored " - f"login could not be used or refreshed. Re-authenticate with " - f"[cyan]helix sandbox login {config.agent.backend}[/cyan], then " - f"[cyan]helix resume[/cyan]. Last report: " - f"{credential_failures.last_message()}" - ) + _display = backend_display_name(config.agent.backend) + if credential_failures.all_transient(): + # Every failure was a lost refresh race: the shared login was + # refreshed by another candidate and is most likely fine. Telling + # the operator to re-login here would throw away a working + # credential and teach them to distrust a healthy run. + print_error( + f"{len(credential_failures)} mutation(s) failed on the shared " + f"{_display} credential, not on their code: {_failed_ids}. " + f"Each lost a refresh race (another candidate refreshed the " + f"shared login first) and failed again on its one retry. The " + f"stored login is most likely usable: run " + f"[cyan]helix resume[/cyan] first, and only re-authenticate " + f"if this keeps recurring. Last report: " + f"{credential_failures.last_message()}" + ) + else: + print_error( + f"{len(credential_failures)} mutation(s) failed on the shared " + f"{_display} credential, not on their code: {_failed_ids}. " + f"The backend reported that its stored login could not be " + f"used or refreshed. Re-authenticate with " + f"[cyan]helix sandbox login {config.agent.backend}[/cyan], " + f"then [cyan]helix resume[/cyan]. Last report: " + f"{credential_failures.last_message()}" + ) best = frontier.best() diff --git a/src/helix/exceptions.py b/src/helix/exceptions.py index 198fb234..778bbb16 100644 --- a/src/helix/exceptions.py +++ b/src/helix/exceptions.py @@ -141,8 +141,16 @@ class CredentialRefreshError(HelixError): continues, but every credential-classified failure is counted and named in the end-of-run summary. Detection is anchored on distinctive wording read out of the shipped backend CLIs; see ``helix.mutator``. + + ``transient`` is True when the backend's wording says the refresh token was + *already used* -- the lost-refresh-race outcome, where another candidate + has just stored a fresh credential. That case is retried once before it is + raised, and when it is raised the operator is pointed at ``helix resume`` + rather than at a re-login the stored credential does not need. """ + transient: bool = False + # --------------------------------------------------------------------------- # Formatted error printing diff --git a/src/helix/mutator.py b/src/helix/mutator.py index 493fe34b..0937431f 100644 --- a/src/helix/mutator.py +++ b/src/helix/mutator.py @@ -619,14 +619,26 @@ def _looks_like_rate_limit(text: str) -> bool: # "auth") is the false-positive trap this repo has already paid for once -- # a candidate whose own diff or test output mentions tokens must never be # reported to the operator as a broken login. +# Markers whose failure is *transient*: the credential is not broken, this +# invocation merely lost a refresh race. Checked before the general markers so +# the more specific wording is what gets reported, and so the caller can retry +# once against the credential the winner has just stored. +_TRANSIENT_CREDENTIAL_FAILURE_MARKERS: tuple[str, ...] = ( + # Codex CLI (codex-cli 0.130.0): the suffix it appends when another + # process spent the single-use refresh token first. The full sentence is + # "Your access token could not be refreshed because your refresh token + # was already used. Please log out and sign in again." -- the "log out" + # advice is the CLI's, and is wrong for this case: the shared auth.json + # already holds the refreshed credential. + "because your refresh token was already used", +) + _CREDENTIAL_FAILURE_MARKERS: tuple[str, ...] = ( # Codex CLI (codex-cli 0.130.0). One prefix covers every suffix the CLI - # appends: "... because your refresh token was already used." / "... has - # expired." / "... was revoked." / "... because you have since logged out - # or signed in to another account." / the bare + # appends: "... has expired." / "... was revoked." / "... because you have + # since logged out or signed in to another account." / the bare # "Your access token could not be refreshed. Please log out and sign in - # again." The already-used variant is the one a lost refresh race - # produces. + # again." (The already-used suffix is matched first, above, as transient.) "your access token could not be refreshed", "failed to refresh token while getting account", "chatgpt account id not available, please re-run `codex login`", @@ -650,12 +662,17 @@ def credential_failure_marker(text: str) -> str | None: if not text: return None lower = text.lower() - for marker in _CREDENTIAL_FAILURE_MARKERS: + for marker in _TRANSIENT_CREDENTIAL_FAILURE_MARKERS + _CREDENTIAL_FAILURE_MARKERS: if marker in lower: return marker return None +def credential_failure_is_transient(marker: str) -> bool: + """True when *marker* names a lost refresh race rather than a dead login.""" + return marker in _TRANSIENT_CREDENTIAL_FAILURE_MARKERS + + def _errored_envelope_texts(parsed: dict[str, Any]) -> list[str]: """Return the message text of every envelope node flagged ``is_error``. @@ -722,10 +739,43 @@ def _credential_refresh_error( cmd_str: str, worktree_path: str, result: subprocess.CompletedProcess[str], + retried: bool = False, ) -> CredentialRefreshError: - return CredentialRefreshError( - f"{backend_name} could not use its stored credential " - f"(matched {marker!r} in {where})", + transient = credential_failure_is_transient(marker) + if transient: + # A lost refresh race. The winner has stored a fresh credential, so + # sending the operator to re-login would discard a working one. + message = ( + f"{backend_name} lost a refresh race on the shared credential " + f"(matched {marker!r} in {where}" + f"{'; failed again on retry' if retried else ''})" + ) + suggestion = ( + f"This is a credential failure, not a failed mutation: another " + f"candidate refreshed the shared {backend_name} login first and " + "the token this invocation held was already spent. The refreshed " + "credential is stored and should be usable" + + ( + ", but a retry with it also failed. Run `helix resume`; the " + "stored login most likely does not need to be redone, so only " + "re-authenticate if this keeps recurring." + if retried + else "." + ) + ) + else: + message = ( + f"{backend_name} could not use its stored credential " + f"(matched {marker!r} in {where})" + ) + suggestion = ( + f"This is a credential failure, not a failed mutation: {backend_name} " + "reported that its stored login could not be used or refreshed. " + f"Re-authenticate with `helix sandbox login {backend}`, then resume " + "the run; nothing is wrong with the candidate's code." + ) + error = CredentialRefreshError( + message, operation=f"{backend_name} invocation", phase="credential check", command=cmd_str, @@ -733,13 +783,10 @@ def _credential_refresh_error( stdout=result.stdout, stderr=result.stderr, exit_code=result.returncode, - suggestion=( - f"This is a credential failure, not a failed mutation: {backend_name} " - "reported that its stored login could not be used or refreshed. " - f"Re-authenticate with `helix sandbox login {backend}`, then resume " - "the run; nothing is wrong with the candidate's code." - ), + suggestion=suggestion, ) + error.transient = transient + return error # --------------------------------------------------------------------------- @@ -1794,49 +1841,97 @@ def invoke_claude_code( backend_env.update( agent_state_env(backend, state_root=str(opencode_state_dir)) ) - if sandbox is not None and sandbox.enabled: - sandbox_image = resolve_sandbox_image(sandbox, backend) - result = run_sandboxed_command( - args, - cwd=worktree_path, - env=backend_env, - sandbox=sandbox, - scope="agent", - sync_back=True, - image=sandbox_image, - agent_backend=backend, - ) - else: - result = subprocess.run( - args, - cwd=worktree_path, - capture_output=True, - text=True, - env=backend_env, - ) - - parsed: dict[str, Any] | None = None - try: - if result.returncode == 0: - parsed = _parse_backend_output( - backend, - result, - cmd_str=cmd_str, - worktree_path=worktree_path, + def _attempt(*, retried: bool) -> tuple[dict[str, Any], UsageStats]: + """Run the backend once and classify the outcome.""" + if sandbox is not None and sandbox.enabled: + sandbox_image = resolve_sandbox_image(sandbox, backend) + result = run_sandboxed_command( + args, + cwd=worktree_path, + env=backend_env, + sandbox=sandbox, + scope="agent", + sync_back=True, + image=sandbox_image, + agent_backend=backend, ) - usage = _normalise_usage_stats(parsed) - # A backend can report an unusable credential and still exit 0 -- - # measured on codex-cli 0.130.0, whose refresh failure is swallowed - # entirely (exit 0, empty stderr, even at RUST_LOG=info). The - # envelope's own ``is_error`` flag is the only signal left on this - # path, so read it here rather than letting the failure pass as a - # successful-but-useless mutation. + else: + result = subprocess.run( + args, + cwd=worktree_path, + capture_output=True, + text=True, + env=backend_env, + ) + + parsed: dict[str, Any] | None = None + try: + if result.returncode == 0: + parsed = _parse_backend_output( + backend, + result, + cmd_str=cmd_str, + worktree_path=worktree_path, + ) + usage = _normalise_usage_stats(parsed) + # A backend can report an unusable credential and still exit 0 -- + # measured on codex-cli 0.130.0, whose refresh failure is swallowed + # entirely (exit 0, empty stderr, even at RUST_LOG=info). The + # envelope's own ``is_error`` flag is the only signal left on this + # path, so read it here rather than letting the failure pass as a + # successful-but-useless mutation. + evidence = _credential_failure_evidence(parsed, result) + if evidence is not None: + marker, where = evidence + logger.error( + "Credential failure detected for %s in %s: matched %r", + backend_name, + where, + marker, + ) + raise _credential_refresh_error( + backend=backend, + backend_name=backend_name, + marker=marker, + where=where, + cmd_str=cmd_str, + worktree_path=worktree_path, + result=result, + retried=retried, + ) + if backend == "claude": + error_text = str(parsed.get("error", "")) + if _looks_like_rate_limit(error_text): + logger.error( + "Rate limit detected in JSON response: %s", error_text[:200] + ) + raise RateLimitError( + f"{backend_name} returned a rate/usage limit error in JSON response", + operation=f"{backend_name} invocation", + phase="JSON parsing", + command=cmd_str, + cwd=str(worktree_path), + stdout=result.stdout, + stderr=result.stderr, + exit_code=result.returncode, + suggestion=( + f"{backend_name} reported a rate limit. " + "Retry after backoff or check your API quota." + ), + ) + return parsed, usage + + # Classify a credential failure ahead of the rate-limit and generic + # paths. The markers are disjoint from the rate-limit keywords, and + # "the login is unusable" is a strictly more actionable verdict than + # "the backend exited non-zero". evidence = _credential_failure_evidence(parsed, result) if evidence is not None: marker, where = evidence logger.error( - "Credential failure detected for %s in %s: matched %r", + "Credential failure detected for %s (exit %d) in %s: matched %r", backend_name, + result.returncode, where, marker, ) @@ -1848,63 +1943,62 @@ def invoke_claude_code( cmd_str=cmd_str, worktree_path=worktree_path, result=result, + retried=retried, + ) + + rate_limit_source = result.stderr or result.stdout + if _looks_like_rate_limit(rate_limit_source): + logger.error( + "Rate limit detected in subprocess exit for %s (code %d): %s", + backend_name, + result.returncode, + rate_limit_source[:200], + ) + raise RateLimitError( + f"{backend_name} hit a rate/usage limit (exit code {result.returncode})", + operation=f"{backend_name} invocation", + phase="subprocess exit", + command=cmd_str, + cwd=str(worktree_path), + stdout=result.stdout, + stderr=result.stderr, + exit_code=result.returncode, + suggestion=( + f"{backend_name} reported a rate limit. " + "Retry after backoff or check your quota." + ), ) + + # Claude's max-turns exhaustion is intentionally treated as partial + # success because the subprocess may have already produced useful edits. if backend == "claude": - error_text = str(parsed.get("error", "")) - if _looks_like_rate_limit(error_text): - logger.error( - "Rate limit detected in JSON response: %s", error_text[:200] - ) - raise RateLimitError( - f"{backend_name} returned a rate/usage limit error in JSON response", - operation=f"{backend_name} invocation", - phase="JSON parsing", - command=cmd_str, - cwd=str(worktree_path), - stdout=result.stdout, - stderr=result.stderr, - exit_code=result.returncode, - suggestion=( - f"{backend_name} reported a rate limit. " - "Retry after backoff or check your API quota." - ), + try: + parsed = _parse_backend_output( + backend, + result, + cmd_str=cmd_str, + worktree_path=worktree_path, ) - return parsed, usage - - # Classify a credential failure ahead of the rate-limit and generic - # paths. The markers are disjoint from the rate-limit keywords, and - # "the login is unusable" is a strictly more actionable verdict than - # "the backend exited non-zero". - evidence = _credential_failure_evidence(parsed, result) - if evidence is not None: - marker, where = evidence - logger.error( - "Credential failure detected for %s (exit %d) in %s: matched %r", - backend_name, - result.returncode, - where, - marker, - ) - raise _credential_refresh_error( - backend=backend, - backend_name=backend_name, - marker=marker, - where=where, + usage = _normalise_usage_stats(parsed) + if parsed.get("subtype") == "error_max_turns": + logger.warning( + "Claude Code reached max_turns limit (%s turns) — treating as partial success.", + parsed.get("num_turns", "?"), + ) + return parsed, usage + except MutationError: + parsed = None + + parsed = _parse_backend_output( + backend, + result, cmd_str=cmd_str, worktree_path=worktree_path, - result=result, ) + usage = _normalise_usage_stats(parsed) - rate_limit_source = result.stderr or result.stdout - if _looks_like_rate_limit(rate_limit_source): - logger.error( - "Rate limit detected in subprocess exit for %s (code %d): %s", - backend_name, - result.returncode, - rate_limit_source[:200], - ) - raise RateLimitError( - f"{backend_name} hit a rate/usage limit (exit code {result.returncode})", + raise MutationError( + f"{backend_name} exited with code {result.returncode}", operation=f"{backend_name} invocation", phase="subprocess exit", command=cmd_str, @@ -1912,60 +2006,34 @@ def invoke_claude_code( stdout=result.stdout, stderr=result.stderr, exit_code=result.returncode, - suggestion=( - f"{backend_name} reported a rate limit. " - "Retry after backoff or check your quota." - ), + suggestion="Check stderr for rate limits, permission errors, or model availability.", + ) + finally: + _write_backend_artifacts( + worktree_path, + backend=backend, + command=cmd_str, + result=result, + parsed=parsed, + sandbox=sandbox, ) - # Claude's max-turns exhaustion is intentionally treated as partial - # success because the subprocess may have already produced useful edits. - if backend == "claude": - try: - parsed = _parse_backend_output( - backend, - result, - cmd_str=cmd_str, - worktree_path=worktree_path, - ) - usage = _normalise_usage_stats(parsed) - if parsed.get("subtype") == "error_max_turns": - logger.warning( - "Claude Code reached max_turns limit (%s turns) — treating as partial success.", - parsed.get("num_turns", "?"), - ) - return parsed, usage - except MutationError: - parsed = None - - parsed = _parse_backend_output( - backend, - result, - cmd_str=cmd_str, - worktree_path=worktree_path, - ) - usage = _normalise_usage_stats(parsed) - - raise MutationError( - f"{backend_name} exited with code {result.returncode}", - operation=f"{backend_name} invocation", - phase="subprocess exit", - command=cmd_str, - cwd=str(worktree_path), - stdout=result.stdout, - stderr=result.stderr, - exit_code=result.returncode, - suggestion="Check stderr for rate limits, permission errors, or model availability.", - ) - finally: - _write_backend_artifacts( - worktree_path, - backend=backend, - command=cmd_str, - result=result, - parsed=parsed, - sandbox=sandbox, + try: + return _attempt(retried=False) + except CredentialRefreshError as exc: + if not exc.transient: + raise + # Lost a refresh race: another candidate has already stored the + # refreshed credential in the shared volume, so a second invocation + # starts from a working login. One retry; a second loss in a row is + # reported as-is rather than looping. + logger.warning( + "%s lost a refresh race on the shared credential; retrying the " + "invocation once against the refreshed credential (%s).", + backend_name, + exc, ) + return _attempt(retried=True) # --------------------------------------------------------------------------- diff --git a/tests/unit/test_credential_failure_classification.py b/tests/unit/test_credential_failure_classification.py index 24974b8f..5eefc5d2 100644 --- a/tests/unit/test_credential_failure_classification.py +++ b/tests/unit/test_credential_failure_classification.py @@ -28,7 +28,11 @@ MutationError, RateLimitError, ) -from helix.mutator import credential_failure_marker, invoke_claude_code +from helix.mutator import ( + credential_failure_is_transient, + credential_failure_marker, + invoke_claude_code, +) # Codex CLI 0.130.0 -- the four suffixes it appends to one prefix, plus the @@ -111,9 +115,31 @@ def test_ordinary_candidate_output_is_not_a_credential_failure( def test_marker_is_returned_as_evidence(self) -> None: """The caller names what matched instead of asserting a bare verdict.""" - marker = credential_failure_marker(CODEX_ALREADY_USED) + marker = credential_failure_marker(CODEX_EXPIRED) assert marker == "your access token could not be refreshed" + def test_already_used_is_the_transient_marker(self) -> None: + """Losing a refresh race is named as such, not as a dead login. + + The already-used suffix is the exact outcome a lost race produces; it + must be matched ahead of the generic prefix so it can be retried and + so the operator is not sent to re-login over a working credential. + """ + marker = credential_failure_marker(CODEX_ALREADY_USED) + assert marker == "because your refresh token was already used" + assert credential_failure_is_transient(marker) + + @pytest.mark.parametrize( + "text", + [CODEX_EXPIRED, CODEX_REVOKED, CODEX_OTHER_ACCOUNT, CODEX_BARE, + CODEX_GET_ACCOUNT, CODEX_NO_ACCOUNT, OPENCODE_REFRESH_FAILED, + CLAUDE_OAUTH_REFRESH, CLAUDE_INVALID_KEY], + ) + def test_every_other_wording_is_not_transient(self, text: str) -> None: + marker = credential_failure_marker(text) + assert marker is not None + assert not credential_failure_is_transient(marker) + def test_embedded_in_a_larger_stream_is_still_found(self) -> None: stream = "\n".join( ["running 3 tests", CODEX_ALREADY_USED, "process exited"] @@ -148,14 +174,15 @@ class TestInvocationClassification: def test_non_zero_exit_with_cli_wording_on_stderr( self, mocker: Any, tmp_path: Path ) -> None: - _patch_backend(mocker, returncode=1, stderr=CODEX_ALREADY_USED) + _patch_backend(mocker, returncode=1, stderr=CODEX_EXPIRED) with pytest.raises(CredentialRefreshError) as exc: invoke_claude_code( str(tmp_path), "p", AgentConfig(backend="codex") ) err = exc.value assert err.exit_code == 1 - assert err.stderr == CODEX_ALREADY_USED + assert err.stderr == CODEX_EXPIRED + assert err.transient is False assert "credential" in err.suggestion.lower() assert "helix sandbox login codex" in err.suggestion @@ -272,3 +299,103 @@ def test_rate_limit_still_wins_its_own_classification( invoke_claude_code( str(tmp_path), "p", AgentConfig(backend="codex") ) + + +def _completed( + returncode: int, stdout: str = "", stderr: str = "" +) -> subprocess.CompletedProcess[str]: + return subprocess.CompletedProcess( + args=["backend"], returncode=returncode, stdout=stdout, stderr=stderr + ) + + +CODEX_SUCCESS_STREAM = json.dumps({"type": "turn.completed"}) + + +class TestLostRefreshRaceIsRetried: + """The already-used outcome is the race this work exists to handle. + + When it happens, the *winner* has just written a refreshed credential to + the shared volume, so the right response is to invoke again against it, + not to drop the slot and tell the operator to redo a login that is fine. + """ + + def test_already_used_is_retried_once_and_the_retry_can_succeed( + self, mocker: Any, tmp_path: Path + ) -> None: + run = mocker.patch( + "helix.mutator.subprocess.run", + side_effect=[ + _completed(1, stderr=CODEX_ALREADY_USED), + _completed(0, stdout=CODEX_SUCCESS_STREAM), + ], + ) + parsed, _usage = invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + assert parsed["events"] + assert run.call_count == 2 + + def test_zero_exit_already_used_envelope_is_retried_too( + self, mocker: Any, tmp_path: Path + ) -> None: + """Codex swallows the failure on exit 0; the envelope path retries as well.""" + stream = "\n".join( + [ + json.dumps({"type": "thread.started"}), + json.dumps({"type": "error", "is_error": True, + "message": CODEX_ALREADY_USED}), + ] + ) + run = mocker.patch( + "helix.mutator.subprocess.run", + side_effect=[ + _completed(0, stdout=stream), + _completed(0, stdout=CODEX_SUCCESS_STREAM), + ], + ) + parsed, _usage = invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + assert parsed["events"] + assert run.call_count == 2 + + def test_second_loss_is_raised_without_a_relogin_instruction( + self, mocker: Any, tmp_path: Path + ) -> None: + run = mocker.patch( + "helix.mutator.subprocess.run", + side_effect=[ + _completed(1, stderr=CODEX_ALREADY_USED), + _completed(1, stderr=CODEX_ALREADY_USED), + ], + ) + with pytest.raises(CredentialRefreshError) as exc: + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + err = exc.value + assert run.call_count == 2 # exactly one retry, no loop + assert err.transient is True + assert "retry" in str(err).lower() + # The stored credential is the winner's fresh one; do not tell the + # operator to throw it away. + assert "sandbox login" not in err.suggestion + assert "helix resume" in err.suggestion + + @pytest.mark.parametrize("text", [CODEX_EXPIRED, CODEX_REVOKED, CODEX_BARE]) + def test_a_dead_login_is_not_retried( + self, mocker: Any, tmp_path: Path, text: str + ) -> None: + """Retrying an expired or revoked credential only burns a turn.""" + run = mocker.patch( + "helix.mutator.subprocess.run", + side_effect=[_completed(1, stderr=text)], + ) + with pytest.raises(CredentialRefreshError) as exc: + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + assert run.call_count == 1 + assert exc.value.transient is False + assert "helix sandbox login codex" in exc.value.suggestion diff --git a/tests/unit/test_credential_warm_loop.py b/tests/unit/test_credential_warm_loop.py index 5509c12c..829e389d 100644 --- a/tests/unit/test_credential_warm_loop.py +++ b/tests/unit/test_credential_warm_loop.py @@ -222,6 +222,44 @@ def run_eval(candidate, config, split=None, instances=None, **kwargs): assert "1 mutation(s) failed on the shared" in out assert "helix sandbox login" in out + def test_lost_refresh_race_does_not_demand_a_relogin( + self, + mocker, # noqa: F811 + tmp_path, + all_mocks, # noqa: F811 + warm_calls, + capsys: pytest.CaptureFixture[str], + ) -> None: + """When every failure was a lost refresh race, the winner has stored a + working credential. The summary must say resume, not re-login.""" + seed = make_candidate("g0-s0") + all_mocks["create_seed_worktree"].return_value = seed + all_mocks["run_evaluator"].side_effect = ( + lambda candidate, *a, **k: make_eval_result( + candidate.id, {"i1": 0.5, "i2": 0.5} + ) + ) + exc = CredentialRefreshError( + "Codex CLI lost a refresh race on the shared credential " + "(matched 'because your refresh token was already used' in " + "stderr; failed again on retry)", + suggestion="Run `helix resume`.", + ) + exc.transient = True + all_mocks["mutate"].side_effect = exc + + config = _sandboxed( + make_config(max_generations=2, perfect_score_threshold=None) + ) + result = run_evolution(config, tmp_path, tmp_path / ".helix") + assert result.best_candidate.id == "g0-s0" + + out = " ".join(capsys.readouterr().out.lower().split()) + assert "credential" in out + assert "refresh race" in out + assert "helix resume" in out + assert "helix sandbox login" not in out + def test_clean_run_says_nothing_about_credentials( self, mocker, # noqa: F811 From ac9da92f55328f4c5a797ca7bc18a29c9c9a2fd8 Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Wed, 9 Sep 2026 17:07:31 -0700 Subject: [PATCH 06/16] fix(sandbox): create the per-candidate agent-state tree mode 0700 opencode.db, which lives in this tree while a candidate runs, carries OAuth access and refresh tokens. The tree already sat under a tempfile.mkdtemp directory (0700) and was removed with it in run_sandboxed_commands' finally, but the state directory itself was created with the umask default, so the privacy guarantee rested on the parent alone. Create it and its per-backend subdirectory 0700 explicitly, document the host-side location and lifetime, and pin both the mode and the removal in tests. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4 --- docs/agent-state-isolation.md | 7 +++++ src/helix/sandbox.py | 10 +++++- tests/unit/test_agent_state.py | 56 ++++++++++++++++++++++++++++++++++ 3 files changed, 72 insertions(+), 1 deletion(-) diff --git a/docs/agent-state-isolation.md b/docs/agent-state-isolation.md index c413eca7..80baf69a 100644 --- a/docs/agent-state-isolation.md +++ b/docs/agent-state-isolation.md @@ -12,6 +12,13 @@ agent state" section covers the three backends that worked. This note records the reasoning for the one that did not, so it does not get re-litigated from scratch. +On the host, `/helix-state` is a directory inside the candidate's +`helix-sandbox-*` scratch tree (a `tempfile.mkdtemp` directory, mode `0700`), +created `0700` in its own right, and removed with that tree as soon as the +candidate's container exits. That matters because part of what lands there is +a credential store -- opencode's `opencode.db` carries OAuth access and refresh +tokens -- so it must never be world-readable or left behind. + All observations below are from the images HELIX ships (`ghcr.io/ke7/helix-evo-runner-*:latest`), against synthetic credentials in throwaway volumes. diff --git a/src/helix/sandbox.py b/src/helix/sandbox.py index 9ae22790..0db9f7b2 100644 --- a/src/helix/sandbox.py +++ b/src/helix/sandbox.py @@ -937,6 +937,13 @@ def _prepare_agent_state_dir( removed by the ``_safe_rmtree`` in :func:`run_sandboxed_commands`. It is chowned to ``node`` because the container runs as that user, matching how the workspace copy is handed over. + + Some of what lands here is a credential store -- opencode's ``opencode.db`` + carries OAuth access and refresh tokens -- and it is on host disk for the + candidate's lifetime. Two things keep it private: *tmp_path* comes from + :func:`tempfile.mkdtemp`, which creates it mode ``0700``, and the state + tree itself is created ``0700`` so the guarantee does not rest on the + parent alone (or on the umask) once ownership passes to ``node``. """ if scope != "agent" or agent_backend is None: return None @@ -944,8 +951,9 @@ def _prepare_agent_state_dir( if not subdirs: return None state_dir = tmp_path / "agent-state" + state_dir.mkdir(mode=0o700, exist_ok=True) for name in subdirs: - (state_dir / name).mkdir(parents=True, exist_ok=True) + (state_dir / name).mkdir(mode=0o700, parents=True, exist_ok=True) _docker_chown_workspace(state_dir, image, "node:node") return state_dir diff --git a/tests/unit/test_agent_state.py b/tests/unit/test_agent_state.py index 9eb66a6d..3532bc6f 100644 --- a/tests/unit/test_agent_state.py +++ b/tests/unit/test_agent_state.py @@ -8,6 +8,8 @@ from __future__ import annotations +import stat +import subprocess from pathlib import Path from unittest.mock import MagicMock @@ -222,6 +224,60 @@ def test_state_dir_lives_in_the_per_candidate_scratch_tree( assert (state_dir / "codex").is_dir() +def test_state_dir_is_private_to_the_owner(tmp_path: Path, mocker) -> None: + """``opencode.db`` holds OAuth tokens and sits on host disk while the + candidate runs; the tree must be 0700 in its own right, not only by + virtue of the mkdtemp parent.""" + mocker.patch("helix.sandbox._docker_chown_workspace") + state_dir = _prepare_agent_state_dir( + tmp_path, scope="agent", agent_backend="opencode", image="img" + ) + assert state_dir is not None + assert stat.S_IMODE(state_dir.stat().st_mode) == 0o700 + assert stat.S_IMODE((state_dir / "opencode").stat().st_mode) == 0o700 + + +def test_state_dir_is_removed_with_the_candidate_scratch_tree( + tmp_path: Path, mocker +) -> None: + """The credential-bearing state must not outlive the candidate.""" + import helix.sandbox as sandbox_mod + + created: list[Path] = [] + real_mkdtemp = sandbox_mod.tempfile.mkdtemp + + def _mkdtemp(**kwargs): + path = real_mkdtemp(dir=tmp_path, **kwargs) + created.append(Path(path)) + return path + + mocker.patch.object(sandbox_mod.tempfile, "mkdtemp", _mkdtemp) + mocker.patch("helix.sandbox._copy_tree_contents") + mocker.patch("helix.sandbox._init_synthetic_git_repo") + mocker.patch("helix.sandbox._docker_chown_workspace") + mocker.patch("helix.sandbox._docker_relax_workspace_permissions") + mocker.patch("helix.sandbox._host_owner", return_value=None) + mocker.patch("helix.sandbox._run_docker") + mocker.patch( + "helix.sandbox._run_docker_process", + return_value=subprocess.CompletedProcess(["docker"], 0, "", ""), + ) + source = tmp_path / "src" + source.mkdir() + sandbox_mod.run_sandboxed_command( + ["true"], + cwd=source, + env={}, + sandbox=SandboxConfig(enabled=True, image="img"), + scope="agent", + sync_back=False, + agent_backend="opencode", + ) + assert len(created) == 1 + assert not (created[0] / "agent-state").exists() + assert not created[0].exists() + + # --------------------------------------------------------------------------- # Backend argv wiring # --------------------------------------------------------------------------- From 18fd2a0f3ae148633825321eac4fd8177568dfd9 Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Wed, 9 Sep 2026 17:07:31 -0700 Subject: [PATCH 07/16] test: deselect docker_integration tests from a bare pytest run tests/integration/ runs real backend containers and is not part of CI, yet a bare `pytest` collected it (skipping when Docker or an image was missing, running real containers otherwise). Deselect the marker by default via addopts; a command-line `-m docker_integration` still opts in. README says how. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_015HBBoDVQK7baNMQBhgRkh4 --- README.md | 6 +++++- pyproject.toml | 5 +++++ 2 files changed, 10 insertions(+), 1 deletion(-) diff --git a/README.md b/README.md index f3bfe4f0..84fc1eca 100644 --- a/README.md +++ b/README.md @@ -670,7 +670,11 @@ if you route `XDG_CONFIG_HOME` through `passthrough_env` or `[env]`, the cursor backend logs a warning because it will break that backend's login. `helix.agent_state.REJECTED_AGENT_STATE_KNOBS` records these so they are not re-tried, and `tests/integration/test_agent_state_isolation.py` pins the -behaviour against the real CLIs. +behaviour against the real CLIs. That suite runs real containers, so a bare +`pytest` does not collect it; opt in with +`pytest -m docker_integration tests/integration/` (set +`HELIX_DOCKER_TESTS_STRICT=1` to fail rather than skip when Docker or an image +is missing). **What still crosses candidates.** Relocation is partial, and the residue is listed per backend in `helix.agent_state.UNRELOCATED_AGENT_STATE`. The diff --git a/pyproject.toml b/pyproject.toml index e1be41b9..f287a1b2 100644 --- a/pyproject.toml +++ b/pyproject.toml @@ -81,6 +81,11 @@ testpaths = ["tests"] # (a now-fixed NB-2-style infinite loop that wedged CI for 24 minutes). timeout = 60 timeout_method = "thread" +# tests/integration/ runs real backend containers and is not run in CI. Keep +# a bare `pytest` from collecting it by default; opt in with +# `pytest -m docker_integration tests/integration/` (a command-line -m +# overrides this). +addopts = "-m 'not docker_integration'" markers = [ "diff_harness: differential-testing harness (phases 2-4)", "docker_integration: runs real containers; needs a Docker daemon and the backend images", From fb59a13a560896c7a2946e9d190f4acf5042cf7e Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Thu, 10 Sep 2026 18:54:01 -0700 Subject: [PATCH 08/16] Cover the agy backend in per-candidate agent state and credential warm main replaced the gemini backend with agy (#71); this branch's per-backend tables still named gemini and lacked agy. - helix.backends.CREDENTIAL_WARM_SKIP_REASONS: drop gemini; add agy. Its registered status probe is a file test on the OAuth token that touches no credential path, `agy models` exits 0 even when logged out and has not been measured for a refresh, and there is no agy credential to measure one against, so it is left unwarmed rather than warmed on a guess. - helix.agent_state.UNRELOCATED_AGENT_STATE: drop gemini; add agy with the residue observed on agy 1.1.27 -- conversations/, conversation_summaries.db, brain/, cache/, history.jsonl, log/, knowledge/, presence/, settings.json, all under ~/.gemini/antigravity-cli/ next to the OAuth token. No knob relocates the state without the credential (ANTIGRAVITY_EXECUTABLE_DATA_DIR is unverified and unused), so agy is not relocated, like claude. - tests: gemini -> agy in test_agent_state and test_credential_warm parametrizations; the docker_integration suite gains AGY_IMAGE and an agy case that pins the no-knob contract and skips with the reason. - docs/agent-state-isolation.md and README: agy row and residue; the README's host-credential paragraph now stays with the auth-volume list it describes instead of trailing the agent-state section. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0126UoDvKj2BN5SHLH81aqnW --- README.md | 16 +++++------ docs/agent-state-isolation.md | 27 ++++++++++++++---- src/helix/agent_state.py | 20 +++++++++++-- src/helix/backends.py | 15 ++++++---- .../integration/test_agent_state_isolation.py | 28 +++++++++++++++++++ tests/unit/test_agent_state.py | 6 ++-- tests/unit/test_credential_warm.py | 2 +- 7 files changed, 88 insertions(+), 26 deletions(-) diff --git a/README.md b/README.md index dd161221..bfbbf2d0 100644 --- a/README.md +++ b/README.md @@ -636,6 +636,14 @@ and complete provider login in one setup session. The volume names are `helix-auth-agy`, `helix-auth-claude`, `helix-auth-codex`, `helix-auth-cursor`, and `helix-auth-opencode`. +This avoids copying host credential stores into Docker. On macOS, Claude/Cursor +browser-login tokens may live in Keychain; on Linux they may live in +Secret Service/libsecret, GNOME Keyring, KWallet, or another desktop keyring. +Those stores are session- and OS-specific, so copying their databases into a +Linux Docker image is not a reliable authentication mechanism. If your +evaluator uses a local proxy, keep that endpoint in your evaluator code as +usual. Docker Desktop supports `host.docker.internal`; Linux users can set +`add_host_gateway = true`. #### Per-candidate agent state @@ -700,14 +708,6 @@ significant cases: the state without also moving the credential: the same all-or-nothing problem as claude. (`ANTIGRAVITY_EXECUTABLE_DATA_DIR` exists in the binary but its semantics are unverified, so it is not used.) -This avoids copying host credential stores into Docker. On macOS, Claude/Cursor -browser-login tokens may live in Keychain; on Linux they may live in -Secret Service/libsecret, GNOME Keyring, KWallet, or another desktop keyring. -Those stores are session- and OS-specific, so copying their databases into a -Linux Docker image is not a reliable authentication mechanism. If your -evaluator uses a local proxy, keep that endpoint in your evaluator code as -usual. Docker Desktop supports `host.docker.internal`; Linux users can set -`add_host_gateway = true`. By default HELIX chooses a published backend-specific mutator image from `agent.backend`: `ghcr.io/ke7/helix-evo-runner-agy`, diff --git a/docs/agent-state-isolation.md b/docs/agent-state-isolation.md index 80baf69a..4b80226e 100644 --- a/docs/agent-state-isolation.md +++ b/docs/agent-state-isolation.md @@ -1,4 +1,4 @@ -# Per-candidate agent state, and why claude is not isolated +# Per-candidate agent state, and why claude and agy are not isolated HELIX mounts one login volume per backend (`helix-auth-`) at `/home/node`, read-write, in every candidate container. That mount is shared on @@ -9,7 +9,7 @@ The problem this document is about is everything *else* the CLIs write into that volume. `helix.agent_state` relocates what it safely can to a per-candidate directory mounted at `/helix-state`; the README's "Per-candidate agent state" section covers the three backends that worked. This note records -the reasoning for the one that did not, so it does not get re-litigated from +the reasoning for the two that did not, so it does not get re-litigated from scratch. On the host, `/helix-state` is a directory inside the candidate's @@ -87,12 +87,27 @@ reports exactly the same thing), but it does mean the only way to prove a claude change is safe is to run it against a live login. Proving isolation by risking the credential it is supposed to protect is a bad trade. +## agy: the same all-or-nothing problem, with no second knob + +Antigravity CLI (`agy`, observed at 1.1.27 on a real install) keeps everything +under `~/.gemini/antigravity-cli/`: `conversations/`, +`conversation_summaries.db`, `brain/`, `cache/`, `history.jsonl`, `log/`, +`knowledge/`, `presence/` and `settings.json` all live in the same directory +as its OAuth token, `antigravity-oauth-token`. No knob is known that relocates +the state without also relocating the credential, which is exactly the +`CLAUDE_CONFIG_DIR` problem above. `ANTIGRAVITY_EXECUTABLE_DATA_DIR` exists in +the binary but its semantics are unverified, so it is not used. agy is not +relocated; its residue is recorded under the `agy` key of +`helix.agent_state.UNRELOCATED_AGENT_STATE`, and +`tests/integration/test_agent_state_isolation.py` skips it with that reason +rather than pretending to verify a knob that does not exist. + ## Conclusion -Claude is left exactly as it is. Its cross-candidate residue is recorded in -`helix.agent_state.UNRELOCATED_AGENT_STATE` under the `claude` key so that it -is discoverable rather than forgotten. Three of four backends are isolated; -this one is documented instead. +Claude and agy are left exactly as they are. Their cross-candidate residue is +recorded in `helix.agent_state.UNRELOCATED_AGENT_STATE` under the `claude` and +`agy` keys so that it is discoverable rather than forgotten. Three of five +backends are isolated; these two are documented instead. Anyone revisiting this should start by re-running the footprint check against the current CLI, because the specific directories named above are version diff --git a/src/helix/agent_state.py b/src/helix/agent_state.py index 64d12747..ac0d7150 100644 --- a/src/helix/agent_state.py +++ b/src/helix/agent_state.py @@ -49,7 +49,7 @@ ) """Backends with a knob that moves state without moving the credential. -``claude`` and ``gemini`` are absent on purpose; see +``claude`` and ``agy`` are absent on purpose; see :data:`UNRELOCATED_AGENT_STATE`. """ @@ -80,7 +80,23 @@ ".claude/backups/", ".claude.json", ), - "gemini": (".gemini/", ".config/google-gemini/"), + # agy (Antigravity CLI 1.1.27) keeps every piece of working state in the + # same directory as its OAuth token, ``.gemini/antigravity-cli/``. No knob + # is known that relocates the state without also relocating the + # credential -- the same all-or-nothing problem as claude. + # ``ANTIGRAVITY_EXECUTABLE_DATA_DIR`` exists in the binary but its + # semantics are unverified, so it is not used. + "agy": ( + ".gemini/antigravity-cli/conversations/", # full transcripts + ".gemini/antigravity-cli/conversation_summaries.db", + ".gemini/antigravity-cli/brain/", + ".gemini/antigravity-cli/cache/", + ".gemini/antigravity-cli/history.jsonl", + ".gemini/antigravity-cli/log/", + ".gemini/antigravity-cli/knowledge/", + ".gemini/antigravity-cli/presence/", + ".gemini/antigravity-cli/settings.json", + ), } diff --git a/src/helix/backends.py b/src/helix/backends.py index d1945d7b..26b7338a 100644 --- a/src/helix/backends.py +++ b/src/helix/backends.py @@ -191,12 +191,15 @@ "an API key instead, so there is no single-use grant for candidates to " "compete over." ), - "gemini": ( - "No free Gemini CLI command is known to take the refresh path. The " - "registered status command is `gemini --version`, which reports the " - "version and touches no credential, so warming with it would be a " - "placebo; and no Gemini credential exists to measure a real refresh " - "against. Left unwarmed deliberately rather than warmed on a guess." + "agy": ( + "No free Antigravity CLI command is known to take the refresh path. " + "The registered status probe is a file test " + "(`test -s ~/.gemini/antigravity-cli/antigravity-oauth-token`), which " + "touches no credential path, so warming with it would be a placebo. " + "`agy models` exits 0 even when logged out, and whether it takes the " + "refresh path has not been measured; no agy credential exists to " + "measure one against. Left unwarmed deliberately rather than warmed " + "on a guess." ), "opencode": ( "OpenCode refreshes an `oauth`-type credential only from inside the " diff --git a/tests/integration/test_agent_state_isolation.py b/tests/integration/test_agent_state_isolation.py index 0c3bf1e2..805993e4 100644 --- a/tests/integration/test_agent_state_isolation.py +++ b/tests/integration/test_agent_state_isolation.py @@ -28,6 +28,7 @@ pytestmark = pytest.mark.docker_integration +AGY_IMAGE = "ghcr.io/ke7/helix-evo-runner-agy:latest" CODEX_IMAGE = "ghcr.io/ke7/helix-evo-runner-codex:latest" CURSOR_IMAGE = "ghcr.io/ke7/helix-evo-runner-cursor:latest" OPENCODE_IMAGE = "ghcr.io/ke7/helix-evo-runner-opencode:latest" @@ -280,6 +281,33 @@ def test_opencode_xdg_data_home_would_hide_the_credential( # --------------------------------------------------------------------------- +# --------------------------------------------------------------------------- +# agy +# --------------------------------------------------------------------------- + + +def test_agy_state_is_not_relocated() -> None: + """agy has no knob to verify; its residue is documented instead. + + Antigravity CLI 1.1.27 keeps its working state (``conversations/``, + ``conversation_summaries.db``, ``brain/``, ``cache/``, ``history.jsonl``, + ``log/``, ``knowledge/``, ``presence/``, ``settings.json``) in + ``~/.gemini/antigravity-cli/``, the same directory as its OAuth token, and + no knob is known that moves the one without the other -- the same + all-or-nothing problem as claude. There is therefore no relocation to + prove in a container (``AGY_IMAGE``); the three assertions this suite + makes would need a knob that does not exist. What *can* be pinned + without a container is that HELIX emits nothing for agy. + """ + assert agent_state_env("agy", state_root=AGENT_STATE_CONTAINER_ROOT) == {} + assert agent_state_cli_args("agy", state_root=AGENT_STATE_CONTAINER_ROOT) == [] + pytest.skip( + "agy exposes no knob that separates its state from its credential " + "(see helix.agent_state.UNRELOCATED_AGENT_STATE['agy']); nothing to " + "verify against the container until one exists" + ) + + def test_real_auth_volumes_are_never_addressed() -> None: """No test in this suite may name a concrete login volume. diff --git a/tests/unit/test_agent_state.py b/tests/unit/test_agent_state.py index 3532bc6f..d2ef5b06 100644 --- a/tests/unit/test_agent_state.py +++ b/tests/unit/test_agent_state.py @@ -66,9 +66,9 @@ def test_cursor_relocates_state_via_config_dir() -> None: assert env == {"CURSOR_CONFIG_DIR": "/helix-state/cursor"} -@pytest.mark.parametrize("backend", ["claude", "gemini"]) +@pytest.mark.parametrize("backend", ["agy", "claude"]) def test_backends_without_a_safe_knob_get_nothing(backend: str) -> None: - """claude and gemini have no knob that separates state from credential.""" + """agy and claude have no knob that separates state from credential.""" assert agent_state_env(backend, state_root=AGENT_STATE_CONTAINER_ROOT) == {} assert agent_state_cli_args(backend, state_root=AGENT_STATE_CONTAINER_ROOT) == [] assert agent_state_subdirs(backend) == () @@ -300,7 +300,7 @@ def test_codex_argv_unchanged_without_a_sandbox() -> None: assert not any("sqlite_home" in a for a in args) -@pytest.mark.parametrize("backend", ["claude", "cursor", "gemini", "opencode"]) +@pytest.mark.parametrize("backend", ["agy", "claude", "cursor", "opencode"]) def test_non_codex_argv_never_carries_a_state_override(backend: str) -> None: args = _build_backend_args( "/workspace", diff --git a/tests/unit/test_credential_warm.py b/tests/unit/test_credential_warm.py index 3d550c0f..ddad6c2f 100644 --- a/tests/unit/test_credential_warm.py +++ b/tests/unit/test_credential_warm.py @@ -37,7 +37,7 @@ WARMED_BACKENDS = ("codex",) -SKIPPED_BACKENDS = ("claude", "cursor", "gemini", "opencode") +SKIPPED_BACKENDS = ("agy", "claude", "cursor", "opencode") def _completed(returncode: int, stderr: str = "") -> subprocess.CompletedProcess[str]: From 9a84db468db3bdaf535fcded82ffb8028c47f175 Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Thu, 10 Sep 2026 19:27:47 -0700 Subject: [PATCH 09/16] Drop per-candidate agent state relocation Remove helix.agent_state and everything that hung off it: the per-candidate /helix-state mount and its 0700 host directory, the codex/opencode/cursor relocation knobs in the sandbox and the mutator argv, the cursor XDG_CONFIG_HOME warning, the isolation docs and their unit and container tests, and the README section. The unsandboxed opencode SQLite isolation keeps the OPENCODE_DB fix (XDG_DATA_HOME hid an existing login); the value is now set inline. The credential warm, credential-failure classification, and the docker_integration opt-in are unchanged. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0126UoDvKj2BN5SHLH81aqnW --- README.md | 62 +--- docs/agent-state-isolation.md | 114 ------ src/helix/agent_state.py | 197 ----------- src/helix/mutator.py | 70 ++-- src/helix/sandbox.py | 60 ---- tests/integration/conftest.py | 2 +- .../integration/test_agent_state_isolation.py | 324 ----------------- tests/unit/test_agent_state.py | 328 ------------------ tests/unit/test_sandbox.py | 11 +- 9 files changed, 29 insertions(+), 1139 deletions(-) delete mode 100644 docs/agent-state-isolation.md delete mode 100644 src/helix/agent_state.py delete mode 100644 tests/integration/test_agent_state_isolation.py delete mode 100644 tests/unit/test_agent_state.py diff --git a/README.md b/README.md index bfbbf2d0..21830c99 100644 --- a/README.md +++ b/README.md @@ -645,70 +645,12 @@ evaluator uses a local proxy, keep that endpoint in your evaluator code as usual. Docker Desktop supports `host.docker.internal`; Linux users can set `add_host_gateway = true`. -#### Per-candidate agent state - -The auth volume is shared by every candidate container on purpose: it is what -lets a CLI refresh its token and take its cross-process refresh lock. But the -CLIs also write their *working state* under `/home/node` — transcripts, session -databases, memories, to-do lists — so without further work candidate N would -start by reading candidate N-1's leftovers. For an optimizer whose candidates -are meant to be independent samples, that contaminates the experiment; some of -that state (opencode's `opencode.db`) also carries token columns. - -HELIX therefore mounts a second, per-candidate directory at `/helix-state` — -deliberately outside `/home/node`, so the shared auth mount is unchanged — and -points each backend's state at it. The directory lives in the same temporary -tree as the sandbox workspace copy, so it is created and deleted with the -candidate. `helix.agent_state` holds the knobs: - -| Backend | Knob | Moves | Credential | -| --- | --- | --- | --- | -| `codex` | `-c sqlite_home=…` | `state_5.sqlite`, `logs_2.sqlite` (+`-wal`/`-shm`) | `.codex/auth.json` stays shared | -| `opencode` | `OPENCODE_DB=…` | `opencode.db` (+`-wal`/`-shm`) | `auth.json` and the lock dir stay shared | -| `cursor` | `CURSOR_CONFIG_DIR=…` | the whole `~/.cursor` tree | `~/.config/cursor/auth.json` stays shared | -| `claude` | none | — | see below | -| `agy` | none | — | see below | - -Picking the knob matters: the obvious environment variable is usually the wrong -one because it relocates the credential too, which silently makes an existing -login invisible. `XDG_DATA_HOME` does this to opencode (`opencode auth list` -then reports `0 credentials`) and `XDG_CONFIG_HOME` does it to cursor -(`cursor-agent status` then reports `Not logged in`). HELIX never sets either; -if you route `XDG_CONFIG_HOME` through `passthrough_env` or `[env]`, the cursor -backend logs a warning because it will break that backend's login. -`helix.agent_state.REJECTED_AGENT_STATE_KNOBS` records these so they are not -re-tried, and `tests/integration/test_agent_state_isolation.py` pins the -behaviour against the real CLIs. That suite runs real containers, so a bare -`pytest` does not collect it; opt in with +The container-backed tests in `tests/integration/` run real backend images, +so a bare `pytest` does not collect them; opt in with `pytest -m docker_integration tests/integration/` (set `HELIX_DOCKER_TESTS_STRICT=1` to fail rather than skip when Docker or an image is missing). -**What still crosses candidates.** Relocation is partial, and the residue is -listed per backend in `helix.agent_state.UNRELOCATED_AGENT_STATE`. The -significant cases: - -- **codex** keeps writing its session rollout transcript to - `.codex/sessions//rollout-*.jsonl`, plus `shell_snapshots/` and - `memories/`, in the shared volume. `sqlite_home` does not cover these and the - CLI exposes no separate knob for them; only `CODEX_HOME` moves them, and that - moves `auth.json` with them. -- **opencode** keeps `log/*.log` and `storage/session_diff/ses_*.json`. -- **claude** is not relocated at all. `CLAUDE_CONFIG_DIR` is all-or-nothing — - it moves `.credentials.json` together with the transcripts — and the - alternative of mounting empty per-candidate volumes over - `.claude/projects`, `.claude/sessions`, `.claude/telemetry` and - `.claude/backups` was evaluated and rejected; see - `docs/agent-state-isolation.md`. -- **agy** is not relocated either. Antigravity CLI 1.1.27 keeps all of its - working state — `conversations/`, `conversation_summaries.db`, `brain/`, - `cache/`, `history.jsonl`, `log/`, `knowledge/`, `presence/`, - `settings.json` — under `~/.gemini/antigravity-cli/`, the same directory as - its OAuth token (`antigravity-oauth-token`), and no knob is known that moves - the state without also moving the credential: the same all-or-nothing - problem as claude. (`ANTIGRAVITY_EXECUTABLE_DATA_DIR` exists in the binary - but its semantics are unverified, so it is not used.) - By default HELIX chooses a published backend-specific mutator image from `agent.backend`: `ghcr.io/ke7/helix-evo-runner-agy`, `ghcr.io/ke7/helix-evo-runner-claude`, diff --git a/docs/agent-state-isolation.md b/docs/agent-state-isolation.md deleted file mode 100644 index 4b80226e..00000000 --- a/docs/agent-state-isolation.md +++ /dev/null @@ -1,114 +0,0 @@ -# Per-candidate agent state, and why claude and agy are not isolated - -HELIX mounts one login volume per backend (`helix-auth-`) at -`/home/node`, read-write, in every candidate container. That mount is shared on -purpose and is not negotiable: it is what lets each CLI refresh its token and -take its cross-process refresh lock across concurrent candidates. - -The problem this document is about is everything *else* the CLIs write into -that volume. `helix.agent_state` relocates what it safely can to a -per-candidate directory mounted at `/helix-state`; the README's "Per-candidate -agent state" section covers the three backends that worked. This note records -the reasoning for the two that did not, so it does not get re-litigated from -scratch. - -On the host, `/helix-state` is a directory inside the candidate's -`helix-sandbox-*` scratch tree (a `tempfile.mkdtemp` directory, mode `0700`), -created `0700` in its own right, and removed with that tree as soon as the -candidate's container exits. That matters because part of what lands there is -a credential store -- opencode's `opencode.db` carries OAuth access and refresh -tokens -- so it must never be world-readable or left behind. - -All observations below are from the images HELIX ships -(`ghcr.io/ke7/helix-evo-runner-*:latest`), against synthetic credentials in -throwaway volumes. - -## Why `CLAUDE_CONFIG_DIR` is not usable - -It is all-or-nothing. It relocates `.credentials.json` together with the -transcripts, and it additionally pulls `.claude.json` into whatever it points -at. Pointing it at a per-candidate directory would give each candidate a clean -state tree and no credential, which defeats the entire purpose of the shared -login volume. There is no second knob: of the `CLAUDE_*` variables the CLI -understands, none relocates state alone. - -## Why masking was evaluated and rejected - -The alternative is *masking*: leave the config directory shared and mount empty -per-candidate volumes over its state subdirectories. This was tried against -Claude Code 2.1.138. It does relocate state — `projects/`, `sessions/`, -`telemetry/`, `backups/` and the contents of `.claude.json` all landed in the -per-candidate directory. It was still rejected, for four reasons. - -**1. The mask list has to be maintained against the CLI, and is already -stale.** The subdirectories a reasonable person would name — `projects/`, -`sessions/`, `todos/` — are not the ones this version writes. There is no -`todos/` at all, and there are two that the obvious list misses: `telemetry/` -and `backups/`, both of which carry per-session identifiers. A mask list that -is already wrong for the currently shipped CLI is the clearest possible -evidence that it will drift again, and each drift is silent: a newly added -state directory simply starts leaking between candidates with nothing to -signal it. - -**2. `.claude.json` is a file outside the config directory.** It lives at -`$HOME/.claude.json`, not under `.claude/`, so masking it needs a *file*-level -bind mount rather than a directory one. The CLI also rewrites it through a -backup-and-replace cycle — a `.claude/backups/.claude.json.backup.` -appears on every run. File bind mounts do not survive an atomic -rename-into-place, so this is a mechanism that works until the day the CLI -changes how it saves that file, and then fails in a way that is hard to -attribute. - -**3. Masking writes to the shared volume, which the isolation work is not -allowed to do.** A bind mount needs its mountpoint to exist, and Docker creates -it inside the volume. Masking the four directories plus `.claude.json` added -five new entries to the shared login volume, including turning `.claude.json` -into a 0-byte file there. The knob-based approach used for codex, cursor and -opencode leaves the shared volume byte-for-byte unchanged; masking cannot. - -**4. It silently breaks transcript preservation.** -`helix.sandbox._copy_claude_transcript_from_auth_volume` recovers the session -transcript by starting a *separate* container that mounts only the auth volume -read-only and copies from `sandbox.claude_transcript_root`. That container does -not carry the agent container's masks, so once `projects/` is masked the -transcript it is looking for is no longer in the volume. The helper's -`[ -f "$src" ] || exit 0` guard means this fails silently: -`preserve_backend_transcripts` would keep reporting success while saving -nothing. - -## The verification gap - -Independently of the above, requirement (c) of this work — *demonstrate the CLI -still reports itself authenticated* — cannot be met for claude without a real -grant. Claude Code validates the credential's shape before reporting status, so -a synthetic credential yields `Not logged in · Please run /login`. That is not -caused by masking (an unmasked container with the same synthetic credential -reports exactly the same thing), but it does mean the only way to prove a -claude change is safe is to run it against a live login. Proving isolation by -risking the credential it is supposed to protect is a bad trade. - -## agy: the same all-or-nothing problem, with no second knob - -Antigravity CLI (`agy`, observed at 1.1.27 on a real install) keeps everything -under `~/.gemini/antigravity-cli/`: `conversations/`, -`conversation_summaries.db`, `brain/`, `cache/`, `history.jsonl`, `log/`, -`knowledge/`, `presence/` and `settings.json` all live in the same directory -as its OAuth token, `antigravity-oauth-token`. No knob is known that relocates -the state without also relocating the credential, which is exactly the -`CLAUDE_CONFIG_DIR` problem above. `ANTIGRAVITY_EXECUTABLE_DATA_DIR` exists in -the binary but its semantics are unverified, so it is not used. agy is not -relocated; its residue is recorded under the `agy` key of -`helix.agent_state.UNRELOCATED_AGENT_STATE`, and -`tests/integration/test_agent_state_isolation.py` skips it with that reason -rather than pretending to verify a knob that does not exist. - -## Conclusion - -Claude and agy are left exactly as they are. Their cross-candidate residue is -recorded in `helix.agent_state.UNRELOCATED_AGENT_STATE` under the `claude` and -`agy` keys so that it is discoverable rather than forgotten. Three of five -backends are isolated; these two are documented instead. - -Anyone revisiting this should start by re-running the footprint check against -the current CLI, because the specific directories named above are version -facts, not stable API. diff --git a/src/helix/agent_state.py b/src/helix/agent_state.py deleted file mode 100644 index ac0d7150..00000000 --- a/src/helix/agent_state.py +++ /dev/null @@ -1,197 +0,0 @@ -"""Per-candidate relocation of agent-CLI state, away from the shared auth volume. - -Why this module exists ----------------------- -HELIX mounts one login volume per backend (``helix-auth-``) at -``/home/node`` read-write in *every* candidate container. That mount is what -keeps the CLIs' token refresh and their cross-process refresh locks working, so -it is deliberately shared and must stay exactly as it is. - -The problem is what else rides along in that volume. Each CLI also writes its -*agent state* under ``$HOME`` -- transcripts, session databases, memories, -to-do lists. Because the volume is shared, candidate N starts life reading -candidate N-1's leftovers. For an evolutionary optimizer whose candidates are -meant to be independent samples, that is contamination of the experiment. A -second, smaller consequence is that some of this state doubles as a credential -store (opencode's ``opencode.db`` carries ``access_token`` / ``refresh_token`` -columns), so it should not be lying around in a shared location either. - -What this module does ---------------------- -For the backends that expose a knob separating *state* from *credential*, it -returns the environment variables and CLI arguments that point state at a -per-candidate directory. The credential file is never named, never moved and -never copied: it keeps living in the shared volume exactly where the CLI put -it. See :data:`UNRELOCATED_AGENT_STATE` for what each backend leaves behind. - -Choosing a knob is not obvious, and the wrong choice silently breaks login. -The knobs below were each verified against the real CLI in a container with a -synthetic credential; the rejected alternatives are recorded in -:data:`REJECTED_AGENT_STATE_KNOBS` so nobody re-tries them. -""" - -from __future__ import annotations - -import json - - -AGENT_STATE_CONTAINER_ROOT = "/helix-state" -"""Container path where the per-candidate state directory is mounted. - -Deliberately *outside* ``/home/node``. Mounting anywhere under ``/home/node`` -would place a mountpoint inside the shared auth volume, which creates a new -entry there -- the one thing the shared mount is not allowed to acquire. -""" - - -STATE_RELOCATING_BACKENDS: frozenset[str] = frozenset( - {"codex", "cursor", "opencode"} -) -"""Backends with a knob that moves state without moving the credential. - -``claude`` and ``agy`` are absent on purpose; see -:data:`UNRELOCATED_AGENT_STATE`. -""" - - -UNRELOCATED_AGENT_STATE: dict[str, tuple[str, ...]] = { - # Paths are relative to ``$HOME`` (the shared auth volume mount point) and - # still carry cross-candidate state after relocation. Named here so the - # residue is discoverable rather than forgotten. - "codex": ( - ".codex/sessions////rollout-*.jsonl", # full transcript - ".codex/shell_snapshots/*.sh", - ".codex/memories/", - ".codex/config.toml", - ".codex/installation_id", - ), - "cursor": (), - "opencode": ( - ".local/share/opencode/log/*.log", - ".local/share/opencode/storage/session_diff/ses_*.json", - ".local/share/opencode/storage/migration", - ".config/opencode/.gitignore", - ), - "claude": ( - ".claude/projects//*.jsonl", # full transcript - ".claude/projects//memory/", - ".claude/sessions/", - ".claude/telemetry/", - ".claude/backups/", - ".claude.json", - ), - # agy (Antigravity CLI 1.1.27) keeps every piece of working state in the - # same directory as its OAuth token, ``.gemini/antigravity-cli/``. No knob - # is known that relocates the state without also relocating the - # credential -- the same all-or-nothing problem as claude. - # ``ANTIGRAVITY_EXECUTABLE_DATA_DIR`` exists in the binary but its - # semantics are unverified, so it is not used. - "agy": ( - ".gemini/antigravity-cli/conversations/", # full transcripts - ".gemini/antigravity-cli/conversation_summaries.db", - ".gemini/antigravity-cli/brain/", - ".gemini/antigravity-cli/cache/", - ".gemini/antigravity-cli/history.jsonl", - ".gemini/antigravity-cli/log/", - ".gemini/antigravity-cli/knowledge/", - ".gemini/antigravity-cli/presence/", - ".gemini/antigravity-cli/settings.json", - ), -} - - -REJECTED_AGENT_STATE_KNOBS: dict[str, str] = { - # Each of these looks like the obvious knob and each one breaks login. - "opencode:XDG_DATA_HOME": ( - "moves opencode.db AND auth.json together; with it set, " - "`opencode auth list` reports 0 credentials" - ), - "cursor:XDG_CONFIG_HOME": ( - "moves cli-config.json AND auth.json together; with it set, " - "`cursor-agent status` reports 'Not logged in'" - ), - "cursor:CURSOR_DATA_DIR": "accepted but relocates nothing; cli-config.json stays in $HOME", - "codex:CODEX_HOME": "moves the state databases AND auth.json together", - "claude:CLAUDE_CONFIG_DIR": ( - "moves the transcripts AND .credentials.json together, and pulls " - ".claude.json in as well" - ), -} - - -def _backend_state_dir(backend: str, state_root: str) -> str: - """Return the per-backend subdirectory of the per-candidate state root.""" - return f"{state_root.rstrip('/')}/{backend}" - - -def agent_state_subdirs(backend: str) -> tuple[str, ...]: - """Return directories to create under the state root before the container runs. - - The CLIs are not uniformly willing to create a missing parent directory for - a relocated database, so HELIX creates them itself and keeps the behaviour - deterministic across backends. Paths are relative to the state root. - """ - if backend not in STATE_RELOCATING_BACKENDS: - return () - return (backend,) - - -def agent_state_env(backend: str, *, state_root: str) -> dict[str, str]: - """Environment variables that point *backend*'s state at a per-candidate dir. - - Returns an empty mapping for backends without a safe knob, so callers can - apply the result unconditionally. - """ - state_dir = _backend_state_dir(backend, state_root) - if backend == "opencode": - # Verified: relocates opencode.db and its -wal/-shm companions alone. - # auth.json stays at $HOME/.local/share/opencode/auth.json and the - # refresh lock stays at $HOME/.local/state/opencode/locks/. - return {"OPENCODE_DB": f"{state_dir}/opencode.db"} - if backend == "cursor": - # Verified: relocates the whole ~/.cursor state tree (cli-config.json, - # agent-cli-state.json, statsig-cache.json, projects//mcp-auth.json). - # The credential is read from ${XDG_CONFIG_HOME||~/.config}/cursor/auth.json, - # which this knob does not affect. - return {"CURSOR_CONFIG_DIR": state_dir} - return {} - - -def agent_state_cli_args(backend: str, *, state_root: str) -> list[str]: - """CLI arguments that point *backend*'s state at a per-candidate dir. - - Used for backends whose only knob is a config override rather than an - environment variable. - """ - if backend == "codex": - # Verified: relocates state_5.sqlite and logs_2.sqlite (plus their - # -wal/-shm companions). auth.json stays at $HOME/.codex/auth.json. - # - # ``-c key=value`` requires a TOML literal on the right-hand side; - # ``json.dumps`` emits a double-quoted string that is also valid TOML - # basic-string syntax, matching how ``model_reasoning_effort`` is - # passed in ``helix.mutator._build_backend_args``. - state_dir = _backend_state_dir(backend, state_root) - return ["-c", f"sqlite_home={json.dumps(state_dir)}"] - return [] - - -def cursor_credential_hazard(backend: str, env: dict[str, str]) -> str | None: - """Return a warning when *env* would hide cursor's shared credential. - - ``cursor-agent`` resolves its credential to - ``${XDG_CONFIG_HOME||~/.config}/cursor/auth.json``. HELIX never sets - ``XDG_CONFIG_HOME`` itself -- the env scrub in ``helix.executor`` is an - allowlist -- but a user can route it through ``passthrough_env`` or the - ``[env]`` table in ``helix.toml``. If they do, cursor stops seeing the - shared login volume entirely and reports "Not logged in", which is worth a - warning rather than a silent authentication failure mid-run. - """ - if backend != "cursor" or "XDG_CONFIG_HOME" not in env: - return None - return ( - "XDG_CONFIG_HOME is set for the cursor backend. Cursor reads its " - "credential from ${XDG_CONFIG_HOME}/cursor/auth.json, so this hides the " - "shared login volume and cursor will report 'Not logged in'. Remove " - "XDG_CONFIG_HOME from passthrough_env / [env] in helix.toml." - ) diff --git a/src/helix/mutator.py b/src/helix/mutator.py index c4f64abc..d4b96f79 100644 --- a/src/helix/mutator.py +++ b/src/helix/mutator.py @@ -10,12 +10,6 @@ from pathlib import Path from typing import Any, Callable -from helix.agent_state import ( - AGENT_STATE_CONTAINER_ROOT, - agent_state_cli_args, - agent_state_env, - cursor_credential_hazard, -) from helix.backends import BACKEND_AUTH_ENV, backend_display_name from helix.display import UsageStats from helix.population import Candidate, EvalResult @@ -943,15 +937,7 @@ def _build_backend_args( worktree_path: str, config: AgentConfig, prompt_artifact_name: str, - agent_state_root: str | None = None, ) -> list[str]: - """Build the backend CLI argv. - - *agent_state_root* is the container path of the per-candidate state - directory when the command runs sandboxed, and ``None`` otherwise. Only - backends whose state knob is a CLI override rather than an environment - variable consume it -- currently just codex. - """ backend = config.backend if backend == "agy": args = [ @@ -1017,12 +1003,6 @@ def _build_backend_args( args.extend( ["-c", f"model_reasoning_effort={json.dumps(config.effort)}"] ) - if agent_state_root is not None: - # Points codex's state databases at the per-candidate directory. - # ``auth.json`` is not affected and stays in the shared volume. - args.extend( - agent_state_cli_args(backend, state_root=agent_state_root) - ) args.append(_prompt_file_instruction(prompt_artifact_name)) return args @@ -1892,49 +1872,49 @@ def invoke_claude_code( return _MUTATOR_OVERRIDE(worktree_path, prompt, config) backend = config.backend backend_name = backend_display_name(backend) - sandbox_enabled = sandbox is not None and sandbox.enabled - backend_worktree_path = "/workspace" if sandbox_enabled else worktree_path + backend_worktree_path = ( + "/workspace" if sandbox is not None and sandbox.enabled else worktree_path + ) args = _build_backend_args( backend_worktree_path, config, prompt_artifact_name, - agent_state_root=AGENT_STATE_CONTAINER_ROOT if sandbox_enabled else None, ) cmd_str = shlex.join(args) backend_env = _scrub_environment( passthrough_env=passthrough_env, fixed_env=fixed_env ) _add_backend_auth_env(backend_env, backend) - if warning := cursor_credential_hazard(backend, backend_env): - logger.warning("%s", warning) - if backend == "opencode" and not sandbox_enabled: + if backend == "opencode" and (sandbox is None or not sandbox.enabled): # Per-candidate SQLite isolation for concurrent opencode subprocesses. # + # OpenCode stores its session database at: + # macOS: ~/Library/Application Support/opencode/opencode.db + # Linux: $XDG_DATA_HOME/opencode/opencode.db (default ~/.local/share/opencode/) + # # When multiple proposals run in parallel (num_parallel_proposals > 1), # every worker spawns a fresh `opencode run` subprocess that issues - # `PRAGMA journal_mode = WAL` against a shared database at startup. + # `PRAGMA journal_mode = WAL` against this shared database at startup. # Concurrent WAL-mode requests on the same file produce: # "Failed to run the query 'PRAGMA journal_mode = WAL'" - # (observed in PR #34 E2E re-verify: g1-s1 lost to this error while - # g1-s2 succeeded). - # - # The knob is OPENCODE_DB, which relocates opencode.db and its - # -wal/-shm companions and nothing else. XDG_DATA_HOME would also - # work for the locking problem but moves auth.json with the database, - # which makes an existing opencode login invisible -- verified against - # the real CLI, where `opencode auth list` then reports 0 credentials. + # (observed in PR #34 E2E re-verify: g1-s1 lost to this error while g1-s2 succeeded). # - # The resulting layout is unchanged from the previous XDG_DATA_HOME - # approach: /.helix_opencode_state/opencode/opencode.db. + # Fix: point OPENCODE_DB at a per-candidate database. OPENCODE_DB + # relocates opencode.db and its -wal/-shm companions and nothing else. + # XDG_DATA_HOME would also solve the locking problem but moves + # auth.json along with the database, which makes an existing opencode + # login invisible -- verified against the real CLI, where + # `opencode auth list` then reports 0 credentials. The on-disk layout + # is unchanged from the earlier XDG_DATA_HOME approach: + # /.helix_opencode_state/opencode/opencode.db + # Each parallel worker gets its own fresh database; no contention. # - # The sandbox branch is excluded because the sandbox applies the same - # knob itself, against a container path; see - # ``helix.sandbox._prepare_agent_state_dir``. - opencode_state_dir = Path(worktree_path) / ".helix_opencode_state" - (opencode_state_dir / backend).mkdir(parents=True, exist_ok=True) - backend_env.update( - agent_state_env(backend, state_root=str(opencode_state_dir)) - ) + # The sandbox branch is excluded: container isolation already provides + # per-candidate filesystem separation, so the knob would be redundant. + opencode_state_dir = Path(worktree_path) / ".helix_opencode_state" / "opencode" + opencode_state_dir.mkdir(parents=True, exist_ok=True) + backend_env["OPENCODE_DB"] = str(opencode_state_dir / "opencode.db") + def _attempt(*, retried: bool) -> tuple[dict[str, Any], UsageStats]: """Run the backend once and classify the outcome.""" if sandbox is not None and sandbox.enabled: diff --git a/src/helix/sandbox.py b/src/helix/sandbox.py index 81c4f765..6ca300ac 100644 --- a/src/helix/sandbox.py +++ b/src/helix/sandbox.py @@ -19,11 +19,6 @@ from pathlib import Path from typing import Literal -from helix.agent_state import ( - AGENT_STATE_CONTAINER_ROOT, - agent_state_env, - agent_state_subdirs, -) from helix.backends import ( BACKEND_AUTH_COMMANDS, DEFAULT_BACKEND_IMAGES, @@ -924,41 +919,6 @@ def sandbox_auth_volume_name(agent_backend: str) -> str: return f"helix-auth-{agent_backend}" -def _prepare_agent_state_dir( - tmp_path: Path, - *, - scope: Literal["agent", "evaluator"], - agent_backend: str | None, - image: str, -) -> Path | None: - """Create the per-candidate agent-state directory, or return ``None``. - - The directory lives inside the same temporary tree as the workspace copy, - so it inherits the sandbox's existing per-candidate scratch lifetime and is - removed by the ``_safe_rmtree`` in :func:`run_sandboxed_commands`. It is - chowned to ``node`` because the container runs as that user, matching how - the workspace copy is handed over. - - Some of what lands here is a credential store -- opencode's ``opencode.db`` - carries OAuth access and refresh tokens -- and it is on host disk for the - candidate's lifetime. Two things keep it private: *tmp_path* comes from - :func:`tempfile.mkdtemp`, which creates it mode ``0700``, and the state - tree itself is created ``0700`` so the guarantee does not rest on the - parent alone (or on the umask) once ownership passes to ``node``. - """ - if scope != "agent" or agent_backend is None: - return None - subdirs = agent_state_subdirs(agent_backend) - if not subdirs: - return None - state_dir = tmp_path / "agent-state" - state_dir.mkdir(mode=0o700, exist_ok=True) - for name in subdirs: - (state_dir / name).mkdir(mode=0o700, parents=True, exist_ok=True) - _docker_chown_workspace(state_dir, image, "node:node") - return state_dir - - def _docker_args( command: list[str], env: dict[str, str], @@ -969,7 +929,6 @@ def _docker_args( agent_backend: str | None, network: str | None = None, container_name: str | None = None, - agent_state_dir: Path | None = None, ) -> list[str]: args = [ "docker", @@ -992,14 +951,6 @@ def _docker_args( if agent_backend is None: raise ValueError("agent_backend is required for sandboxed agent commands") args.extend(["-v", f"{sandbox_auth_volume_name(agent_backend)}:/home/node:rw"]) - # The auth volume above is shared across candidates on purpose: it is - # what keeps token refresh and the CLIs' refresh locks working. The - # state mount below is per-candidate and lives outside /home/node, so - # nothing here changes how the credential is shared. - if agent_state_dir is not None: - args.extend( - ["-v", f"{agent_state_dir}:{AGENT_STATE_CONTAINER_ROOT}:rw"] - ) if sandbox.pids_limit is not None: args.extend(["--pids-limit", str(sandbox.pids_limit)]) @@ -1021,10 +972,6 @@ def _docker_args( container_env["PATH"] = ( "/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin" ) - if agent_state_dir is not None and agent_backend is not None: - container_env.update( - agent_state_env(agent_backend, state_root=AGENT_STATE_CONTAINER_ROOT) - ) for key, value in container_env.items(): args.extend(["-e", f"{key}={value}"]) @@ -1071,12 +1018,6 @@ def run_sandboxed_commands( ) _init_synthetic_git_repo(workspace) _docker_chown_workspace(workspace, docker_image, "node:node") - agent_state_dir = _prepare_agent_state_dir( - tmp_path, - scope=scope, - agent_backend=agent_backend, - image=docker_image, - ) sidecar_runtime = ( current_evaluator_sidecar_runtime() if scope == "evaluator" else None ) @@ -1097,7 +1038,6 @@ def run_sandboxed_commands( agent_backend, sidecar_runtime.network if sidecar_runtime is not None else None, container_name=container_name, - agent_state_dir=agent_state_dir, ) try: results.append( diff --git a/tests/integration/conftest.py b/tests/integration/conftest.py index 49ba7197..be1c91ee 100644 --- a/tests/integration/conftest.py +++ b/tests/integration/conftest.py @@ -17,7 +17,7 @@ REAL_AUTH_VOLUME_PREFIX = "helix-auth-" -TEST_VOLUME_PREFIX = "helix-agent-state-test-" +TEST_VOLUME_PREFIX = "helix-integration-test-" def _strict() -> bool: diff --git a/tests/integration/test_agent_state_isolation.py b/tests/integration/test_agent_state_isolation.py deleted file mode 100644 index 805993e4..00000000 --- a/tests/integration/test_agent_state_isolation.py +++ /dev/null @@ -1,324 +0,0 @@ -"""Container proof that agent state relocates and the credential does not. - -Each test asserts the same three things against a real backend container: - -(a) the backend's state lands in the per-candidate directory; -(b) nothing new lands in the shared login volume; -(c) the CLI still reports itself authenticated. - -Credentials are synthetic and live in throwaway volumes (see ``conftest``); -no test logs in, and none can reach a real ``helix-auth-*`` volume. -""" - -from __future__ import annotations - -import re -import subprocess -from pathlib import Path - -import pytest - -from helix.agent_state import ( - AGENT_STATE_CONTAINER_ROOT, - agent_state_cli_args, - agent_state_env, -) - - -pytestmark = pytest.mark.docker_integration - - -AGY_IMAGE = "ghcr.io/ke7/helix-evo-runner-agy:latest" -CODEX_IMAGE = "ghcr.io/ke7/helix-evo-runner-codex:latest" -CURSOR_IMAGE = "ghcr.io/ke7/helix-evo-runner-cursor:latest" -OPENCODE_IMAGE = "ghcr.io/ke7/helix-evo-runner-opencode:latest" - -# Deliberately malformed-but-well-shaped values. They are never accepted by a -# real API; they only have to be present for a CLI to report a stored login. -SYNTHETIC_CODEX_AUTH = ( - 'mkdir -p /home/node/.codex; printf "%s" ' - "'{\"OPENAI_API_KEY\":\"sk-SYNTHETIC-NOT-A-REAL-KEY\"}' " - "> /home/node/.codex/auth.json" -) -SYNTHETIC_CURSOR_AUTH = ( - 'mkdir -p /home/node/.config/cursor; printf "%s" ' - "'{\"accessToken\":\"SYNTHETIC\",\"refreshToken\":\"SYNTHETIC\"}' " - "> /home/node/.config/cursor/auth.json" -) -SYNTHETIC_OPENCODE_AUTH = ( - 'mkdir -p /home/node/.local/share/opencode; printf "%s" ' - "'{\"anthropic\":{\"type\":\"api\",\"key\":\"sk-ant-SYNTHETIC\"}}' " - "> /home/node/.local/share/opencode/auth.json" -) - - -def _run_backend( - *, - image: str, - volume: str, - state_dir: Path, - backend: str, - shell_command: str, - timeout: int = 120, -) -> str: - """Run one backend container the way ``helix.sandbox`` would. - - The auth volume is mounted read-write at ``/home/node`` exactly as in - production; the per-candidate state directory is a separate mount outside - it, carrying the relocation env vars from ``helix.agent_state``. - """ - (state_dir / backend).mkdir(parents=True, exist_ok=True) - args = [ - "docker", - "run", - "--rm", - "--network", - "none", - "--security-opt", - "no-new-privileges", - "--user", - "node", - "-v", - f"{volume}:/home/node:rw", - "-v", - f"{state_dir}:{AGENT_STATE_CONTAINER_ROOT}:rw", - "-e", - "HOME=/home/node", - ] - for key, value in agent_state_env( - backend, state_root=AGENT_STATE_CONTAINER_ROOT - ).items(): - args.extend(["-e", f"{key}={value}"]) - args.extend([image, "sh", "-lc", shell_command]) - result = subprocess.run( - args, capture_output=True, text=True, check=False, timeout=timeout - ) - return result.stdout + result.stderr - - -def _relative_paths(state_dir: Path) -> set[str]: - return { - str(p.relative_to(state_dir)) - for p in state_dir.rglob("*") - if p.is_file() - } - - -# --------------------------------------------------------------------------- -# codex -# --------------------------------------------------------------------------- - - -@pytest.mark.timeout(300) -def test_codex_state_databases_relocate( - tmp_path: Path, require_image, throwaway_volume, volume_listing -) -> None: - image = require_image(CODEX_IMAGE) - volume = throwaway_volume(image, SYNTHETIC_CODEX_AUTH) - before = volume_listing(volume, image) - - state_dir = tmp_path / "agent-state" - sqlite_args = " ".join( - agent_state_cli_args("codex", state_root=AGENT_STATE_CONTAINER_ROOT) - ) - _run_backend( - image=image, - volume=volume, - state_dir=state_dir, - backend="codex", - # The state databases are opened during startup, well before any API - # call, so a short timeout is enough and no model is ever reached - # (the container has no network). - shell_command=( - "cd /tmp; timeout 20 codex exec --json " - f"--dangerously-bypass-approvals-and-sandbox {sqlite_args} hi " - ">/dev/null 2>&1 || true" - ), - ) - - # (a) state landed per-candidate - relocated = _relative_paths(state_dir) - assert any(name.endswith("state_5.sqlite") for name in relocated), relocated - assert any(name.endswith("logs_2.sqlite") for name in relocated), relocated - - # (b) no sqlite state landed in the shared volume - after = volume_listing(volume, image) - new_entries = after - before - assert not [e for e in new_entries if ".sqlite" in e], new_entries - - # (c) the credential is untouched and still reported - status = _run_backend( - image=image, - volume=volume, - state_dir=state_dir, - backend="codex", - shell_command="codex login status", - ) - assert "Logged in" in status, status - - -# --------------------------------------------------------------------------- -# cursor -# --------------------------------------------------------------------------- - - -@pytest.mark.timeout(300) -def test_cursor_state_relocates_and_login_survives( - tmp_path: Path, require_image, throwaway_volume, volume_listing -) -> None: - image = require_image(CURSOR_IMAGE) - volume = throwaway_volume(image, SYNTHETIC_CURSOR_AUTH) - before = volume_listing(volume, image) - - state_dir = tmp_path / "agent-state" - status = _run_backend( - image=image, - volume=volume, - state_dir=state_dir, - backend="cursor", - shell_command="timeout 60 cursor-agent status", - ) - - # (a) ~/.cursor state landed per-candidate - assert "cursor/cli-config.json" in _relative_paths(state_dir) - - # (b) the shared volume is byte-for-byte unchanged - assert volume_listing(volume, image) == before - - # (c) the shared credential is still found - assert "Logged in" in status, status - - -def test_cursor_xdg_config_home_would_hide_the_credential( - tmp_path: Path, require_image, throwaway_volume -) -> None: - """Guard the rejected knob: XDG_CONFIG_HOME breaks cursor's login. - - This is why ``helix.agent_state`` uses CURSOR_CONFIG_DIR and why - ``cursor_credential_hazard`` warns when a user routes XDG_CONFIG_HOME - through ``passthrough_env``. - """ - image = require_image(CURSOR_IMAGE) - volume = throwaway_volume(image, SYNTHETIC_CURSOR_AUTH) - state_dir = tmp_path / "agent-state" - state_dir.mkdir() - - result = subprocess.run( - [ - "docker", "run", "--rm", "--network", "none", "--user", "node", - "-v", f"{volume}:/home/node:rw", - "-v", f"{state_dir}:{AGENT_STATE_CONTAINER_ROOT}:rw", - "-e", "HOME=/home/node", - "-e", f"XDG_CONFIG_HOME={AGENT_STATE_CONTAINER_ROOT}/cursor", - image, "sh", "-lc", "timeout 60 cursor-agent status", - ], - capture_output=True, text=True, check=False, timeout=120, - ) - assert "Not logged in" in result.stdout + result.stderr - - -# --------------------------------------------------------------------------- -# opencode -# --------------------------------------------------------------------------- - - -@pytest.mark.timeout(300) -def test_opencode_database_relocates_and_credential_stays( - tmp_path: Path, require_image, throwaway_volume, volume_listing -) -> None: - image = require_image(OPENCODE_IMAGE) - volume = throwaway_volume(image, SYNTHETIC_OPENCODE_AUTH) - before = volume_listing(volume, image) - - state_dir = tmp_path / "agent-state" - listing = _run_backend( - image=image, - volume=volume, - state_dir=state_dir, - backend="opencode", - shell_command="cd /tmp; timeout 120 opencode auth list", - timeout=200, - ) - - # (a) the database (which also carries token columns) landed per-candidate - assert "opencode/opencode.db" in _relative_paths(state_dir) - - # (b) no database landed in the shared volume - new_entries = volume_listing(volume, image) - before - assert not [e for e in new_entries if "opencode.db" in e], new_entries - - # (c) the shared auth.json is still the credential source, and is seen - assert "auth.json" in listing - assert "0 credentials" not in listing, listing - - -@pytest.mark.timeout(300) -def test_opencode_xdg_data_home_would_hide_the_credential( - tmp_path: Path, require_image, throwaway_volume -) -> None: - """Guard the rejected knob: XDG_DATA_HOME moves auth.json with the database.""" - image = require_image(OPENCODE_IMAGE) - volume = throwaway_volume(image, SYNTHETIC_OPENCODE_AUTH) - state_dir = tmp_path / "agent-state" - state_dir.mkdir() - - result = subprocess.run( - [ - "docker", "run", "--rm", "--network", "none", "--user", "node", - "-v", f"{volume}:/home/node:rw", - "-v", f"{state_dir}:{AGENT_STATE_CONTAINER_ROOT}:rw", - "-e", "HOME=/home/node", - "-e", f"XDG_DATA_HOME={AGENT_STATE_CONTAINER_ROOT}", - image, "sh", "-lc", "cd /tmp; timeout 120 opencode auth list", - ], - capture_output=True, text=True, check=False, timeout=200, - ) - assert "0 credentials" in result.stdout + result.stderr - - -# --------------------------------------------------------------------------- -# The invariant that outranks all of the above -# --------------------------------------------------------------------------- - - -# --------------------------------------------------------------------------- -# agy -# --------------------------------------------------------------------------- - - -def test_agy_state_is_not_relocated() -> None: - """agy has no knob to verify; its residue is documented instead. - - Antigravity CLI 1.1.27 keeps its working state (``conversations/``, - ``conversation_summaries.db``, ``brain/``, ``cache/``, ``history.jsonl``, - ``log/``, ``knowledge/``, ``presence/``, ``settings.json``) in - ``~/.gemini/antigravity-cli/``, the same directory as its OAuth token, and - no knob is known that moves the one without the other -- the same - all-or-nothing problem as claude. There is therefore no relocation to - prove in a container (``AGY_IMAGE``); the three assertions this suite - makes would need a knob that does not exist. What *can* be pinned - without a container is that HELIX emits nothing for agy. - """ - assert agent_state_env("agy", state_root=AGENT_STATE_CONTAINER_ROOT) == {} - assert agent_state_cli_args("agy", state_root=AGENT_STATE_CONTAINER_ROOT) == [] - pytest.skip( - "agy exposes no knob that separates its state from its credential " - "(see helix.agent_state.UNRELOCATED_AGENT_STATE['agy']); nothing to " - "verify against the container until one exists" - ) - - -def test_real_auth_volumes_are_never_addressed() -> None: - """No test in this suite may name a concrete login volume. - - Checked by inspecting the suite source rather than the daemon, so it holds - even on a machine that has no login volumes at all. Prose mentioning the - volume family in the abstract is fine; a resolvable name is not. - """ - concrete_name = re.compile(r"helix-auth-[a-z0-9]+") - for path in (Path(__file__), Path(__file__).parent / "conftest.py"): - for number, line in enumerate(path.read_text().splitlines(), start=1): - if concrete_name.search(line): - pytest.fail( - f"{path.name}:{number} names a real auth volume: {line.strip()}" - ) diff --git a/tests/unit/test_agent_state.py b/tests/unit/test_agent_state.py deleted file mode 100644 index d2ef5b06..00000000 --- a/tests/unit/test_agent_state.py +++ /dev/null @@ -1,328 +0,0 @@ -"""Tests: per-candidate agent-state relocation away from the shared auth volume. - -The invariant these tests defend is narrow and load-bearing: HELIX may move a -backend's *state* to a per-candidate location, but it must never move, copy, -name or shadow the *credential*, because the shared ``helix-auth-`` -volume is what keeps token refresh and the CLIs' refresh locks working. -""" - -from __future__ import annotations - -import stat -import subprocess -from pathlib import Path -from unittest.mock import MagicMock - -import pytest - -from helix.agent_state import ( - AGENT_STATE_CONTAINER_ROOT, - REJECTED_AGENT_STATE_KNOBS, - STATE_RELOCATING_BACKENDS, - UNRELOCATED_AGENT_STATE, - agent_state_cli_args, - agent_state_env, - agent_state_subdirs, - cursor_credential_hazard, -) -from helix.backends import BACKENDS -from helix.config import AgentConfig, SandboxConfig -from helix.mutator import _build_backend_args -from helix.sandbox import _docker_args, _prepare_agent_state_dir - - -# --------------------------------------------------------------------------- -# The state root must never sit inside the shared auth volume -# --------------------------------------------------------------------------- - - -def test_state_root_is_outside_the_auth_volume_mount() -> None: - """The per-candidate mount must not be nested under ``/home/node``. - - Mounting inside the auth volume would create a new entry in it, which is - exactly what the shared mount is not allowed to acquire. - """ - assert not AGENT_STATE_CONTAINER_ROOT.startswith("/home/node") - assert Path(AGENT_STATE_CONTAINER_ROOT).is_absolute() - - -# --------------------------------------------------------------------------- -# Per-backend knobs -# --------------------------------------------------------------------------- - - -def test_codex_relocates_state_databases_via_sqlite_home() -> None: - args = agent_state_cli_args("codex", state_root=AGENT_STATE_CONTAINER_ROOT) - assert args == ["-c", 'sqlite_home="/helix-state/codex"'] - - -def test_opencode_relocates_only_the_database_file() -> None: - env = agent_state_env("opencode", state_root=AGENT_STATE_CONTAINER_ROOT) - assert env == {"OPENCODE_DB": "/helix-state/opencode/opencode.db"} - - -def test_cursor_relocates_state_via_config_dir() -> None: - env = agent_state_env("cursor", state_root=AGENT_STATE_CONTAINER_ROOT) - assert env == {"CURSOR_CONFIG_DIR": "/helix-state/cursor"} - - -@pytest.mark.parametrize("backend", ["agy", "claude"]) -def test_backends_without_a_safe_knob_get_nothing(backend: str) -> None: - """agy and claude have no knob that separates state from credential.""" - assert agent_state_env(backend, state_root=AGENT_STATE_CONTAINER_ROOT) == {} - assert agent_state_cli_args(backend, state_root=AGENT_STATE_CONTAINER_ROOT) == [] - assert agent_state_subdirs(backend) == () - assert backend not in STATE_RELOCATING_BACKENDS - - -@pytest.mark.parametrize("backend", sorted(STATE_RELOCATING_BACKENDS)) -def test_relocating_backends_emit_exactly_one_knob(backend: str) -> None: - """Each backend uses one knob, so there is a single thing to re-verify.""" - knobs = list(agent_state_env(backend, state_root=AGENT_STATE_CONTAINER_ROOT)) - knobs += agent_state_cli_args(backend, state_root=AGENT_STATE_CONTAINER_ROOT)[:1] - assert len(knobs) == 1, f"{backend} should relocate state with one knob" - - -# --------------------------------------------------------------------------- -# Credential-safety: the knobs we must never emit -# --------------------------------------------------------------------------- - - -@pytest.mark.parametrize("backend", BACKENDS) -def test_no_backend_ever_receives_a_credential_moving_knob(backend: str) -> None: - """Regression guard for the knobs recorded in REJECTED_AGENT_STATE_KNOBS. - - Every name below relocates the backend's credential file along with its - state. Emitting any of them would make an existing login invisible to the - CLI, which is the failure this whole module exists to avoid. - """ - forbidden = { - "XDG_DATA_HOME", - "XDG_CONFIG_HOME", - "XDG_STATE_HOME", - "HOME", - "CODEX_HOME", - "CLAUDE_CONFIG_DIR", - "OPENCODE_CONFIG_DIR", - } - env = agent_state_env(backend, state_root=AGENT_STATE_CONTAINER_ROOT) - assert forbidden.isdisjoint(env), ( - f"{backend} must not receive a credential-relocating env var" - ) - rendered = " ".join( - agent_state_cli_args(backend, state_root=AGENT_STATE_CONTAINER_ROOT) - ) - assert "codex_home" not in rendered - - -def test_rejected_knobs_are_documented_with_a_reason() -> None: - """Keep the 'do not re-try this' list honest and non-empty.""" - assert REJECTED_AGENT_STATE_KNOBS - for name, reason in REJECTED_AGENT_STATE_KNOBS.items(): - assert ":" in name, f"{name} should read as 'backend:KNOB'" - assert reason.strip() - - -def test_every_backend_has_a_leftover_state_entry() -> None: - """Residue must be recorded for all backends, including the ones we fixed.""" - assert set(UNRELOCATED_AGENT_STATE) == set(BACKENDS) - - -# --------------------------------------------------------------------------- -# cursor's XDG_CONFIG_HOME hazard -# --------------------------------------------------------------------------- - - -def test_cursor_hazard_warns_when_xdg_config_home_is_present() -> None: - warning = cursor_credential_hazard("cursor", {"XDG_CONFIG_HOME": "/somewhere"}) - assert warning is not None - assert "XDG_CONFIG_HOME" in warning - - -def test_cursor_hazard_silent_when_absent_or_other_backend() -> None: - assert cursor_credential_hazard("cursor", {"PATH": "/usr/bin"}) is None - assert cursor_credential_hazard("codex", {"XDG_CONFIG_HOME": "/x"}) is None - - -# --------------------------------------------------------------------------- -# Sandbox wiring -# --------------------------------------------------------------------------- - - -def _agent_docker_args(backend: str, state_dir: Path | None) -> list[str]: - return _docker_args( - ["echo", "hi"], - {}, - Path("/tmp/workspace"), - SandboxConfig(enabled=True), - "agent", - "img:latest", - backend, - agent_state_dir=state_dir, - ) - - -def test_shared_auth_volume_mount_is_unchanged_by_relocation() -> None: - """The credential mount must stay ``:/home/node:rw``, always.""" - for backend in BACKENDS: - with_state = _agent_docker_args(backend, Path("/tmp/state")) - without_state = _agent_docker_args(backend, None) - expected = f"helix-auth-{backend}:/home/node:rw" - assert expected in with_state - assert expected in without_state - # Relocation adds mounts; it never removes or rewrites the auth mount. - assert with_state.count(expected) == without_state.count(expected) == 1 - - -def test_state_dir_is_mounted_outside_the_auth_volume() -> None: - args = _agent_docker_args("codex", Path("/tmp/state")) - assert f"/tmp/state:{AGENT_STATE_CONTAINER_ROOT}:rw" in args - # No mount target may be nested inside the shared volume. - targets = [ - args[i + 1].split(":")[1] for i, a in enumerate(args) if a == "-v" - ] - nested = [t for t in targets if t.startswith("/home/node/")] - assert not nested, f"mounts nested inside the auth volume: {nested}" - - -def test_relocation_env_reaches_the_container() -> None: - args = _agent_docker_args("cursor", Path("/tmp/state")) - assert f"CURSOR_CONFIG_DIR={AGENT_STATE_CONTAINER_ROOT}/cursor" in args - args = _agent_docker_args("opencode", Path("/tmp/state")) - assert f"OPENCODE_DB={AGENT_STATE_CONTAINER_ROOT}/opencode/opencode.db" in args - - -def test_evaluator_scope_gets_no_state_dir(tmp_path: Path) -> None: - """Only agent commands touch the auth volume, so only they need relocation.""" - assert ( - _prepare_agent_state_dir( - tmp_path, scope="evaluator", agent_backend="codex", image="img" - ) - is None - ) - - -def test_no_state_dir_for_backends_without_a_knob(tmp_path: Path) -> None: - assert ( - _prepare_agent_state_dir( - tmp_path, scope="agent", agent_backend="claude", image="img" - ) - is None - ) - - -def test_state_dir_lives_in_the_per_candidate_scratch_tree( - tmp_path: Path, mocker -) -> None: - """The directory must sit under the sandbox temp tree that is rmtree'd.""" - mocker.patch("helix.sandbox._docker_chown_workspace") - state_dir = _prepare_agent_state_dir( - tmp_path, scope="agent", agent_backend="codex", image="img" - ) - assert state_dir is not None - assert state_dir.is_relative_to(tmp_path) - assert (state_dir / "codex").is_dir() - - -def test_state_dir_is_private_to_the_owner(tmp_path: Path, mocker) -> None: - """``opencode.db`` holds OAuth tokens and sits on host disk while the - candidate runs; the tree must be 0700 in its own right, not only by - virtue of the mkdtemp parent.""" - mocker.patch("helix.sandbox._docker_chown_workspace") - state_dir = _prepare_agent_state_dir( - tmp_path, scope="agent", agent_backend="opencode", image="img" - ) - assert state_dir is not None - assert stat.S_IMODE(state_dir.stat().st_mode) == 0o700 - assert stat.S_IMODE((state_dir / "opencode").stat().st_mode) == 0o700 - - -def test_state_dir_is_removed_with_the_candidate_scratch_tree( - tmp_path: Path, mocker -) -> None: - """The credential-bearing state must not outlive the candidate.""" - import helix.sandbox as sandbox_mod - - created: list[Path] = [] - real_mkdtemp = sandbox_mod.tempfile.mkdtemp - - def _mkdtemp(**kwargs): - path = real_mkdtemp(dir=tmp_path, **kwargs) - created.append(Path(path)) - return path - - mocker.patch.object(sandbox_mod.tempfile, "mkdtemp", _mkdtemp) - mocker.patch("helix.sandbox._copy_tree_contents") - mocker.patch("helix.sandbox._init_synthetic_git_repo") - mocker.patch("helix.sandbox._docker_chown_workspace") - mocker.patch("helix.sandbox._docker_relax_workspace_permissions") - mocker.patch("helix.sandbox._host_owner", return_value=None) - mocker.patch("helix.sandbox._run_docker") - mocker.patch( - "helix.sandbox._run_docker_process", - return_value=subprocess.CompletedProcess(["docker"], 0, "", ""), - ) - source = tmp_path / "src" - source.mkdir() - sandbox_mod.run_sandboxed_command( - ["true"], - cwd=source, - env={}, - sandbox=SandboxConfig(enabled=True, image="img"), - scope="agent", - sync_back=False, - agent_backend="opencode", - ) - assert len(created) == 1 - assert not (created[0] / "agent-state").exists() - assert not created[0].exists() - - -# --------------------------------------------------------------------------- -# Backend argv wiring -# --------------------------------------------------------------------------- - - -def test_codex_argv_carries_sqlite_home_when_sandboxed() -> None: - args = _build_backend_args( - "/workspace", - AgentConfig(backend="codex"), - "prompt.md", - agent_state_root=AGENT_STATE_CONTAINER_ROOT, - ) - assert "-c" in args - assert 'sqlite_home="/helix-state/codex"' in args - - -def test_codex_argv_unchanged_without_a_sandbox() -> None: - """Unsandboxed runs have no container state mount to point at.""" - args = _build_backend_args("/wt", AgentConfig(backend="codex"), "prompt.md") - assert not any("sqlite_home" in a for a in args) - - -@pytest.mark.parametrize("backend", ["agy", "claude", "cursor", "opencode"]) -def test_non_codex_argv_never_carries_a_state_override(backend: str) -> None: - args = _build_backend_args( - "/workspace", - AgentConfig(backend=backend), - "prompt.md", - agent_state_root=AGENT_STATE_CONTAINER_ROOT, - ) - assert not any("sqlite_home" in a for a in args) - - -def test_local_opencode_db_stays_in_the_gitignored_state_dir( - tmp_path: Path, mocker -) -> None: - """Unsandboxed opencode keeps its database inside .helix_opencode_state/.""" - from helix.mutator import invoke_claude_code - - mock_run = mocker.patch("helix.mutator.subprocess.run") - mock_run.return_value = MagicMock( - stdout='{"type":"result","sessionID":"ses_abc"}\n', stderr="", returncode=0 - ) - invoke_claude_code(str(tmp_path), "prompt", AgentConfig(backend="opencode")) - - db_path = Path(mock_run.call_args[1]["env"]["OPENCODE_DB"]) - assert db_path.is_relative_to(tmp_path / ".helix_opencode_state") - assert db_path.parent.is_dir(), "parent dir must exist before opencode starts" diff --git a/tests/unit/test_sandbox.py b/tests/unit/test_sandbox.py index 6feee342..640a2148 100644 --- a/tests/unit/test_sandbox.py +++ b/tests/unit/test_sandbox.py @@ -136,18 +136,9 @@ def fake_run(args, **kwargs): assert "helix-auth-codex:/home/node:rw" in docker_call assert f"{tmp_path}:" not in joined assert "/workspace:rw" in joined - # codex relocates its state databases, so the agent container also gets a - # per-candidate state mount -- outside /home/node, leaving the shared auth - # mount asserted above untouched. - assert "/helix-state:rw" in joined - assert "/home/node/helix-state" not in joined - # Three housekeeping chowns: the workspace before the run, the - # per-candidate state directory before the run (it must be writable by the - # container's ``node`` user), and the workspace again afterwards. chown_calls = [call for call in calls if _is_workspace_chown(call)] - assert len(chown_calls) == 3 + assert len(chown_calls) == 2 assert "node:node" in chown_calls[0] - assert "node:node" in chown_calls[1] def test_evaluator_scope_does_not_mount_agent_auth(tmp_path: Path, mocker): From b046fd243f722978a28a846be2fdec38ad96e55e Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Thu, 10 Sep 2026 19:28:48 -0700 Subject: [PATCH 10/16] Start every candidate with a fresh agent session Probed 2026-09-10 against the real CLIs (plant a fact in one one-shot session, ask for it in a fresh one, same cwd, no tools): only Claude Code recalled it, through auto-memory keyed by repo root, which spans worktrees and the shared sandbox HOME. CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 stops it. codex has a stable `memories` feature flag, off by default; `-c features.memories=false` pins it off. agy, cursor and opencode read nothing from a prior session. BACKEND_FRESH_SESSION_ENV carries the claude switch and is applied next to the auth env in invoke_claude_code (the sandbox forwards every backend env key); the codex switch is part of the fixed argv. Transcripts and session databases stay in the login volume. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0126UoDvKj2BN5SHLH81aqnW --- CHANGELOG.md | 6 ++++++ README.md | 8 ++++++++ src/helix/backends.py | 22 ++++++++++++++++++++++ src/helix/mutator.py | 12 +++++++++++- tests/unit/test_mutator.py | 28 ++++++++++++++++++++++++++++ 5 files changed, 75 insertions(+), 1 deletion(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 5b519c82..4644528d 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -7,6 +7,12 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] +### Added +- Every sandboxed candidate now starts with a fresh agent session: + `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` for `claude` and + `-c features.memories=false` for `codex` (the other backends read nothing + from a prior session); transcripts stay in the `helix-auth-` volume. + ### Changed - **BREAKING**: Removed the `gemini` mutation backend and replaced it with `agy` (Google's Antigravity CLI). Configs with `agent.backend = "gemini"` diff --git a/README.md b/README.md index 21830c99..b0568ff9 100644 --- a/README.md +++ b/README.md @@ -645,6 +645,14 @@ evaluator uses a local proxy, keep that endpoint in your evaluator code as usual. Docker Desktop supports `host.docker.internal`; Linux users can set `add_host_gateway = true`. +Every candidate starts with a fresh agent session: HELIX sets +`CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` for `claude` (its auto-memory is keyed by +repo root and would otherwise span candidates) and passes +`-c features.memories=false` to `codex`; `agy`, `cursor`, and `opencode` were +probed and read nothing from a prior session, so they need no switch. +Transcripts and session databases remain in the `helix-auth-` volume, +so operators can read them after a run. + The container-backed tests in `tests/integration/` run real backend images, so a bare `pytest` does not collect them; opt in with `pytest -m docker_integration tests/integration/` (set diff --git a/src/helix/backends.py b/src/helix/backends.py index 26b7338a..0dfeca52 100644 --- a/src/helix/backends.py +++ b/src/helix/backends.py @@ -73,6 +73,28 @@ "opencode": ("OPENCODE_API_KEY", "OPENAI_API_KEY", "ANTHROPIC_API_KEY"), } +# Environment that makes every candidate start from a fresh agent session. +# +# Candidates share one login volume per backend (mounted at /home/node), so a +# CLI that reads memory from an earlier session would carry state from +# candidate N-1 into candidate N. Probed 2026-09-10 against the real CLIs: +# plant a fact in one one-shot session, ask for it in a fresh one, same cwd, +# no tools. +# claude recalled it -- auto-memory, keyed by repo root, so it spans +# worktrees and the shared HOME; CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 +# stops it and no memory directory is created. +# codex did not; its ``memories`` feature flag is off by default and is +# pinned off in argv (``-c features.memories=false``, see +# ``helix.mutator._build_backend_args``) because it is a config +# override, not an environment variable. +# agy, cursor, opencode did not; they read nothing from a prior session, +# so there is nothing to disable and no entry here. +# Transcripts and session databases are still written to the login volume so +# they can be read after a run. +BACKEND_FRESH_SESSION_ENV: dict[str, dict[str, str]] = { + "claude": {"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1"}, +} + BACKEND_AUTH_COMMANDS: dict[str, dict[str, list[str]]] = { "agy": { # No dedicated non-interactive login subcommand; the bare interactive diff --git a/src/helix/mutator.py b/src/helix/mutator.py index d4b96f79..d81da9cd 100644 --- a/src/helix/mutator.py +++ b/src/helix/mutator.py @@ -10,7 +10,11 @@ from pathlib import Path from typing import Any, Callable -from helix.backends import BACKEND_AUTH_ENV, backend_display_name +from helix.backends import ( + BACKEND_AUTH_ENV, + BACKEND_FRESH_SESSION_ENV, + backend_display_name, +) from helix.display import UsageStats from helix.population import Candidate, EvalResult from helix.config import AgentConfig, HelixConfig, SandboxConfig @@ -990,6 +994,11 @@ def _build_backend_args( "exec", "--json", "--dangerously-bypass-approvals-and-sandbox", + # Pin codex's cross-session memory off (default off, but a flag) + # so no candidate reads a prior candidate's session; see + # ``helix.backends.BACKEND_FRESH_SESSION_ENV``. + "-c", + "features.memories=false", ] if config.model: args.extend(["--model", config.model]) @@ -1885,6 +1894,7 @@ def invoke_claude_code( passthrough_env=passthrough_env, fixed_env=fixed_env ) _add_backend_auth_env(backend_env, backend) + backend_env.update(BACKEND_FRESH_SESSION_ENV.get(backend, {})) if backend == "opencode" and (sandbox is None or not sandbox.enabled): # Per-candidate SQLite isolation for concurrent opencode subprocesses. # diff --git a/tests/unit/test_mutator.py b/tests/unit/test_mutator.py index d78a090c..98ebb3de 100644 --- a/tests/unit/test_mutator.py +++ b/tests/unit/test_mutator.py @@ -845,6 +845,34 @@ def test_codex_effort_config_value_is_json_quoted(self, mocker): args_list = mock_run.call_args[0][0] assert 'model_reasoning_effort="high\\"quoted"' in args_list + def test_codex_cli_args_pin_memories_off(self, mocker): + mock_run = mocker.patch("helix.mutator.subprocess.run") + mock_run.return_value = MagicMock(stdout="", stderr="", returncode=0) + + invoke_claude_code("/tmp/wt", "the prompt", AgentConfig(backend="codex")) + + args_list = mock_run.call_args[0][0] + idx = args_list.index("features.memories=false") + assert args_list[idx - 1] == "-c" + + def test_claude_env_disables_auto_memory(self, mocker): + mock_run = mocker.patch("helix.mutator.subprocess.run") + mock_run.return_value = MagicMock(stdout="{}", stderr="", returncode=0) + + invoke_claude_code("/tmp/wt", "the prompt", AgentConfig(backend="claude")) + + env = mock_run.call_args[1]["env"] + assert env["CLAUDE_CODE_DISABLE_AUTO_MEMORY"] == "1" + + def test_fresh_session_env_not_injected_for_other_backends(self, mocker): + mock_run = mocker.patch("helix.mutator.subprocess.run") + mock_run.return_value = MagicMock(stdout="", stderr="", returncode=0) + + invoke_claude_code("/tmp/wt", "the prompt", AgentConfig(backend="codex")) + + env = mock_run.call_args[1]["env"] + assert "CLAUDE_CODE_DISABLE_AUTO_MEMORY" not in env + def test_cursor_cli_args_use_stream_json(self, mocker): mock_run = mocker.patch("helix.mutator.subprocess.run") mock_run.return_value = MagicMock( From eda879fe3ab76e57a1a950bc6194a05f4525a765 Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Thu, 10 Sep 2026 21:08:22 -0700 Subject: [PATCH 11/16] sandbox: create the per-candidate OpenCode state dir in the workspace copy Every sandboxed agent container mounts the one helix-auth-opencode volume at /home/node with HOME forced, so without a per-candidate OPENCODE_DB all concurrent opencode candidates open the same opencode.db and lose to "PRAGMA journal_mode = WAL". The database will live under the workspace copy (/workspace/.helix_opencode_state/opencode/opencode.db); SQLite does not create parent directories and .helix* paths are excluded from the copy, so the sandbox now creates the directory before the container starts and lists it in the synthetic repo's local excludes. The directory name moves to helix.sandbox so mutator can share it. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0126UoDvKj2BN5SHLH81aqnW --- src/helix/sandbox.py | 33 +++++++++++++++++ tests/unit/test_sandbox.py | 75 ++++++++++++++++++++++++++++++++++++++ 2 files changed, 108 insertions(+) diff --git a/src/helix/sandbox.py b/src/helix/sandbox.py index 6ca300ac..020a61f9 100644 --- a/src/helix/sandbox.py +++ b/src/helix/sandbox.py @@ -38,6 +38,13 @@ "helix_batch.json", } +#: Per-candidate OpenCode state directory (holds ``opencode/opencode.db``). +#: Under the worktree when unsandboxed; created in the per-candidate workspace +#: copy (mounted at ``/workspace``) when sandboxed. Starts with ``.helix`` so +#: it is excluded from the sandbox copy and sync-back like every other HELIX +#: artifact, and ``helix.mutator._ignore_helix_artifacts`` gitignores it. +OPENCODE_STATE_DIR_NAME = ".helix_opencode_state" + @dataclass(frozen=True) class EvaluatorSidecarRuntime: @@ -477,6 +484,30 @@ def _init_synthetic_git_repo(workspace: Path) -> None: ) +def _prepare_opencode_state_dir(workspace: Path) -> None: + """Create the per-candidate OpenCode database directory in *workspace*. + + ``invoke_claude_code`` points ``OPENCODE_DB`` at + ``/workspace/.helix_opencode_state/opencode/opencode.db`` for sandboxed + opencode runs (every container shares one ``/home/node``, so the default + location would be one database for all concurrent candidates). SQLite + does not create parent directories, and ``.helix*`` paths are excluded + from the workspace copy, so the directory has to be made here. It is + listed in the synthetic repo's local excludes so the agent's ``git + status`` never shows the database as an untracked file. + """ + (workspace / OPENCODE_STATE_DIR_NAME / "opencode").mkdir( + parents=True, exist_ok=True + ) + exclude = workspace / ".git" / "info" / "exclude" + try: + exclude.parent.mkdir(parents=True, exist_ok=True) + with exclude.open("a", encoding="utf-8") as fh: + fh.write(f"{OPENCODE_STATE_DIR_NAME}/\n") + except OSError as exc: + logger.debug("could not write %s: %s", exclude, exc) + + def _run_workspace_helper( workspace: Path, image: str, @@ -1017,6 +1048,8 @@ def run_sandboxed_commands( omit_paths=omit_paths, ) _init_synthetic_git_repo(workspace) + if scope == "agent" and agent_backend == "opencode": + _prepare_opencode_state_dir(workspace) _docker_chown_workspace(workspace, docker_image, "node:node") sidecar_runtime = ( current_evaluator_sidecar_runtime() if scope == "evaluator" else None diff --git a/tests/unit/test_sandbox.py b/tests/unit/test_sandbox.py index 640a2148..7c36ce47 100644 --- a/tests/unit/test_sandbox.py +++ b/tests/unit/test_sandbox.py @@ -1117,3 +1117,78 @@ def test_captured_output_is_left_verbatim(self, mocker) -> None: result = sandbox_module._run_docker(args, check=False) assert result.stderr == traceback_text + + +def test_sandboxed_opencode_workspace_carries_its_own_state_dir( + tmp_path: Path, mocker +): + """The database directory named by ``OPENCODE_DB`` must exist inside the + workspace copy before the container starts (SQLite does not create + parent directories), and must not show up as an untracked file to the + agent.""" + source = tmp_path / "candidate" + source.mkdir() + (source / "main.py").write_text("print('hi')\n") + + seen: dict[str, object] = {} + + def fake_run(args, **kwargs): + if args[:2] == ["docker", "run"] and "--user" in args and "-e" in args: + mount = next(a for a in args if a.endswith(":/workspace:rw")) + workspace = Path(mount.split(":")[0]) + seen["state_dir_exists"] = ( + workspace / ".helix_opencode_state" / "opencode" + ).is_dir() + exclude = workspace / ".git" / "info" / "exclude" + seen["excluded"] = exclude.is_file() and ( + ".helix_opencode_state/" in exclude.read_text() + ) + seen["env"] = [a for a in args if a.startswith("OPENCODE_DB=")] + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + + mocker.patch("helix.sandbox.subprocess.run", side_effect=fake_run) + mocker.patch("helix.sandbox._host_owner", return_value="1000:1000") + + run_sandboxed_command( + ["opencode", "run", "prompt"], + cwd=source, + env={"OPENCODE_DB": "/workspace/.helix_opencode_state/opencode/opencode.db"}, + sandbox=SandboxConfig(enabled=True, image="helix-test:latest"), + scope="agent", + sync_back=True, + agent_backend="opencode", + ) + + assert seen["state_dir_exists"] is True + assert seen["excluded"] is True + assert seen["env"] == [ + "OPENCODE_DB=/workspace/.helix_opencode_state/opencode/opencode.db" + ] + # Nothing came back to the candidate's worktree. + assert not (source / ".helix_opencode_state").exists() + + +def test_other_backends_get_no_opencode_state_dir(tmp_path: Path, mocker): + source = tmp_path / "candidate" + source.mkdir() + seen: dict[str, object] = {} + + def fake_run(args, **kwargs): + if args[:2] == ["docker", "run"] and "--user" in args and "-e" in args: + mount = next(a for a in args if a.endswith(":/workspace:rw")) + workspace = Path(mount.split(":")[0]) + seen["state_dir_exists"] = (workspace / ".helix_opencode_state").exists() + return subprocess.CompletedProcess(args, 0, stdout="", stderr="") + + mocker.patch("helix.sandbox.subprocess.run", side_effect=fake_run) + mocker.patch("helix.sandbox._host_owner", return_value="1000:1000") + run_sandboxed_command( + ["codex", "exec", "prompt"], + cwd=source, + env={}, + sandbox=SandboxConfig(enabled=True, image="helix-test:latest"), + scope="agent", + sync_back=False, + agent_backend="codex", + ) + assert seen["state_dir_exists"] is False From 586e6b363adae30dc2c08ca05fcacc5499c69d3a Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Thu, 10 Sep 2026 21:09:05 -0700 Subject: [PATCH 12/16] mutator: read credential failures from structured events; retry a lost race from a fresh worktree Evidence (review items 1, 2): codex 'exec --json' never emits is_error; its failures are {"type":"error","message"} and turn.failed.error.message (codex-rs exec_events.rs), Claude's error envelope carries an 'errors' list, and opencode re-emits session.error as {"type":"error","error": {name,data:{message}}}. _structured_error_texts reads exactly those fields on every exit code, plus stderr on a non-zero exit; the raw JSONL stdout (tool output, agent prose) is never scanned. The non-zero order is Claude error_max_turns partial success first, then credential evidence, then rate limit, then the generic MutationError. The evidence/raise block is one closure, the dead 'or usage' is gone, and marker pinning to CLI prose is a documented risk. Retry (items 6, 7, 8): the one-shot retry after a transient lost race moves out of invoke_claude_code into invoke_with_refresh_race_retry, driven by mutate() and merge(): the retry gets a fresh clone_candidate worktree with the usual prepare_worktree instead of re-running the prompt on a tree holding attempt 1's half-applied edits (sync_back copies them; the sandbox baseline commit hides them). Attempt 1's artifacts are kept as *.attempt1.* (gitignored), the final .helix_backend_result.json records attempts=2, retry_of and the combined usage, and the recovered race is reported through on_refresh_race_recovered so the summary can show it. mutate()'s credential path now records usage like its siblings; any non-HelixError from the retry (TimeoutExpired, OSError) still hands attempt 1's usage to the sink and both callers remove the live worktree on any exception. Item 9 (mutator side): sandboxed opencode sets OPENCODE_DB under the workspace copy; the "container isolation already separates it" comment was false. BACKEND_FRESH_SESSION_ENV is applied with setdefault so an operator's explicit [env] value wins. Tests: real codex/claude/opencode error-event fixtures replace the invented is_error shape; exit 1 after a command_execution printed "Token refresh failed" stays a MutationError; Claude max_turns exit 1 with auth wording in result prose stays partial success; the retry suite runs through mutate() (fresh worktree, both attempts' artifacts, combined usage, TimeoutExpired on the retry) plus one real-git case. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0126UoDvKj2BN5SHLH81aqnW --- src/helix/merger.py | 81 +- src/helix/mutator.py | 858 ++++++++++++------ .../test_credential_failure_classification.py | 782 ++++++++++++---- tests/unit/test_merger.py | 59 ++ tests/unit/test_mutator.py | 84 ++ 5 files changed, 1373 insertions(+), 491 deletions(-) diff --git a/src/helix/merger.py b/src/helix/merger.py index 40f69e97..070af5c7 100644 --- a/src/helix/merger.py +++ b/src/helix/merger.py @@ -5,7 +5,7 @@ import math import random from pathlib import Path -from typing import Callable, Mapping +from typing import Any, Callable, Mapping from helix.display import UsageStats from helix.population import Candidate, EvalResult @@ -17,7 +17,13 @@ RateLimitError, print_helix_error, ) -from helix.mutator import invoke_claude_code, AUTONOMOUS_SYSTEM_PROMPT, _turn_budget_section +from helix.backends import backend_display_name +from helix.mutator import ( # noqa: F401 + AUTONOMOUS_SYSTEM_PROMPT, + _turn_budget_section, + invoke_claude_code, + invoke_with_refresh_race_retry, +) # --------------------------------------------------------------------------- # Merge-acceptance subsample selection (GEPA parity) @@ -240,6 +246,7 @@ def merge( prepare_worktree: Callable[[Candidate], None] | None = None, ancestor: Candidate | None = None, record_usage: Callable[[UsageStats], None] | None = None, + on_refresh_race_recovered: Callable[[str], None] | None = None, ) -> Candidate | None: """Merge *candidate_a* and *candidate_b* using Claude Code. @@ -308,17 +315,25 @@ def merge( Optional sink called exactly once with the backend's token usage, whether or not the merge produced a usable candidate — the same contract as :func:`helix.mutator.mutate`'s parameter of that name. + on_refresh_race_recovered: + Optional sink told when the merge lost a refresh race on the shared + credential and succeeded on its one retry from a fresh worktree; the + same contract as :func:`helix.mutator.mutate`'s parameter. Returns ------- Candidate | None The merged candidate on success, or ``None`` on failure. """ - child = clone_candidate(candidate_a, new_id, base_dir) - child.operation = "merge" - child.parent_ids = [candidate_a.id, candidate_b.id] - if prepare_worktree is not None: - prepare_worktree(child) + def _fresh_child() -> Candidate: + fresh = clone_candidate(candidate_a, new_id, base_dir) + fresh.operation = "merge" + fresh.parent_ids = [candidate_a.id, candidate_b.id] + if prepare_worktree is not None: + prepare_worktree(fresh) + return fresh + + child = _fresh_child() # Diff-rendering mode selection. ``ancestor`` available → compute # the two ancestor-relative diffs that drive the GEPA-style @@ -347,14 +362,39 @@ def merge( diff_b_from_ancestor=diff_b_from_ancestor, ) - try: - _, usage = invoke_claude_code( - child.worktree_path, + def _invoke( + target: Candidate, retried: bool + ) -> tuple[dict[str, Any], UsageStats]: + return invoke_claude_code( + target.worktree_path, prompt, config.agent, passthrough_env=config.passthrough_env, fixed_env=config.env, sandbox=config.sandbox, + retried=retried, + ) + + def _replace(fresh: Candidate) -> None: + nonlocal child + child = fresh + + def _discard_child() -> None: + try: + remove_worktree(child) + except Exception: + pass + + try: + child, usage = invoke_with_refresh_race_retry( + child, + backend_name=backend_display_name(config.agent.backend), + invoke=_invoke, + fresh_child=_fresh_child, + remove_child=remove_worktree, + on_child_replaced=_replace, + record_usage=record_usage, + on_refresh_race_recovered=on_refresh_race_recovered, ) child.usage = usage if record_usage is not None: @@ -366,19 +406,13 @@ def merge( record_usage(exc.usage) exc.operation = f"merge {new_id} ({candidate_a.id} + {candidate_b.id})" print_helix_error(exc) - try: - remove_worktree(child) - except Exception: - pass + _discard_child() return None except RateLimitError as exc: # Rate limit — clean up orphaned worktree, then re-raise. if record_usage is not None and exc.usage is not None: record_usage(exc.usage) - try: - remove_worktree(child) - except Exception: - pass + _discard_child() raise except CredentialRefreshError as exc: # The stored login, not this merge, is what failed. Mirror @@ -391,10 +425,13 @@ def merge( if record_usage is not None and exc.usage is not None: record_usage(exc.usage) exc.operation = f"merge {new_id} ({candidate_a.id} + {candidate_b.id})" - try: - remove_worktree(child) - except Exception: - pass + _discard_child() + raise + except Exception: + # A non-HELIX exception (sandbox ``TimeoutExpired``, ``OSError``) + # carries no usage; an earlier attempt's spend has already reached + # the sink. Do not leak the worktree on the way out. + _discard_child() raise # NOTE: snapshot_candidate() is intentionally NOT called here. diff --git a/src/helix/mutator.py b/src/helix/mutator.py index d81da9cd..373992ec 100644 --- a/src/helix/mutator.py +++ b/src/helix/mutator.py @@ -6,7 +6,9 @@ import logging import os import shlex +import shutil import subprocess +import tempfile from pathlib import Path from typing import Any, Callable @@ -28,7 +30,11 @@ ) from helix.executor import _scrub_environment from helix.lines import split_lf_lines -from helix.sandbox import resolve_sandbox_image, run_sandboxed_command +from helix.sandbox import ( + OPENCODE_STATE_DIR_NAME, + resolve_sandbox_image, + run_sandboxed_command, +) from helix.worktree import clone_candidate, snapshot_candidate, remove_worktree # noqa: F401 logger = logging.getLogger(__name__) @@ -633,6 +639,12 @@ def _looks_like_rate_limit(text: str) -> bool: "because your refresh token was already used", ) +# KNOWN RISK: every marker is pinned to the prose of a specific CLI release +# (versions noted per entry), while the sandbox images install the unpinned +# CLI package and are rebuilt on a schedule. A wording change upstream turns +# a credential failure back into an unclassified one -- it never produces a +# false positive. No test reads real CLI output; when a marker stops matching, +# the symptom is a generic MutationError whose stderr carries the new wording. _CREDENTIAL_FAILURE_MARKERS: tuple[str, ...] = ( # Codex CLI (codex-cli 0.130.0). One prefix covers every suffix the CLI # appends: "... has expired." / "... was revoked." / "... because you have @@ -673,60 +685,139 @@ def credential_failure_is_transient(marker: str) -> bool: return marker in _TRANSIENT_CREDENTIAL_FAILURE_MARKERS -def _errored_envelope_texts(parsed: dict[str, Any]) -> list[str]: - """Return the message text of every envelope node flagged ``is_error``. +#: Event types under which the JSONL backends report a failure of the +#: invocation itself, as opposed to a tool call the agent made. Read from the +#: shipped CLIs' own event definitions: +#: +#: codex ``{"type": "error", "message": ...}`` and +#: ``{"type": "turn.failed", "error": {"message": ...}}`` +#: (codex-rs ``exec/src/exec_events.rs``: ``ThreadErrorEvent``, +#: ``TurnFailedEvent``). ``codex exec --json`` never emits an +#: ``is_error`` key. +#: opencode ``{"type": "error", ..., "error": {"name": ..., "data": +#: {"message": ...}}}`` -- the ``session.error`` event that +#: ``opencode run --format json`` re-emits (``cli/cmd/run.ts``). +#: cursor no documented failure event; a ``type: "error"`` line carrying +#: ``message`` / ``error`` is read the same way. +_STRUCTURED_ERROR_EVENT_TYPES: frozenset[str] = frozenset( + {"error", "turn.failed", "session.error"} +) + - ``is_error`` is the backends' own "this node reports a failure" flag: it is - top-level on Claude Code's JSON result envelope and per-event inside the - JSONL streams the other backends emit. Reading it is what lets a - credential failure be recognised on a *zero* exit, where there is no exit - code to key off. +def _error_field_texts(value: Any) -> list[str]: + """Flatten an ``error``-shaped field into the message strings it carries. - The flag alone never classifies anything. A ``tool_result`` carrying - ``is_error`` is usually just the agent's own failing shell command -- an - ordinary code failure. It only narrows the text that - :func:`credential_failure_marker` is then asked about, so a classification - still needs the backend to have said, in its own words, that the - credential is unusable. + Accepts the shapes the backends use: a bare string, a list of strings + (Claude Code's ``errors``), or an object with ``message`` / ``error`` / + ``data.message`` (opencode's ``{name, data: {message}}``). Anything else + contributes nothing -- an unknown shape is not evidence. """ - texts: list[str] = [] - for node in _walk_json(parsed): - flag = node.get("is_error") - if flag is not True: - continue - for key in ("error", "message", "result", "error_message", "text", "content"): - value = node.get(key) - if isinstance(value, str) and value: - texts.append(value) + if isinstance(value, str): + return [value] if value else [] + if isinstance(value, list): + texts: list[str] = [] + for item in value: + texts.extend(_error_field_texts(item)) + return texts + if isinstance(value, dict): + texts = [] + for key in ("message", "error"): + texts.extend(_error_field_texts(value.get(key))) + data = value.get("data") + if isinstance(data, dict): + texts.extend(_error_field_texts(data.get("message"))) + return texts + return [] + + +def _structured_error_texts( + backend: str, parsed: dict[str, Any] | None, stdout: str +) -> list[str]: + """Return every failure the backend reported through a *structured* field. + + This is the only stdout-derived text the credential classifier is ever + asked about. It is deliberately narrow: the fields read here are the ones + the CLI itself uses to say "this invocation failed", never the agent's + prose and never a tool's captured output. A candidate whose test suite + prints ``Token refresh failed: 400`` into a ``command_execution`` item, or + whose final message quotes a 401, must not be able to talk HELIX into + declaring the operator's login dead. + + JSONL backends (codex, cursor, opencode): the ``message`` / ``error`` of + events whose type is in :data:`_STRUCTURED_ERROR_EVENT_TYPES`. When the + strict parse has not happened yet (non-zero exit), the stream is read + leniently so a malformed line elsewhere does not hide the error event. + + Envelope backends (claude, agy): the envelope's ``errors`` list and + ``error`` field always, and ``result`` only when the envelope flags + ``is_error`` -- on a successful turn ``result`` is the assistant's prose. + """ + if backend in {"codex", "cursor", "opencode"}: + events: Any = parsed.get("events") if parsed is not None else None + if not isinstance(events, list): + events = _parse_jsonl_output( + stdout, + backend=backend, + cmd_str="", + worktree_path="", + stderr="", + exit_code=0, + strict=False, + )["events"] + texts: list[str] = [] + for event in events: + if not isinstance(event, dict): + continue + if event.get("type") not in _STRUCTURED_ERROR_EVENT_TYPES: + continue + texts.extend(_error_field_texts(event.get("message"))) + texts.extend(_error_field_texts(event.get("error"))) + return texts + + envelope: dict[str, Any] | None = parsed + if envelope is None: + try: + loaded = json.loads(stdout) if stdout.strip() else None + except (json.JSONDecodeError, ValueError, RecursionError): + loaded = None + if isinstance(loaded, dict): + envelope = loaded + if envelope is None: + return [] + texts = [] + texts.extend(_error_field_texts(envelope.get("errors"))) + texts.extend(_error_field_texts(envelope.get("error"))) + if envelope.get("is_error") is True: + texts.extend(_error_field_texts(envelope.get("result"))) return texts def _credential_failure_evidence( + backend: str, parsed: dict[str, Any] | None, result: subprocess.CompletedProcess[str], ) -> tuple[str, str] | None: """Return ``(marker, where)`` when this invocation is a credential failure. - On a **zero** exit only ``is_error``-flagged envelope text is considered. - The run reported success, so the raw streams are full of the agent's own - work; scanning them would let a candidate that merely *edited* an OAuth - code path talk HELIX into declaring the operator's login broken. + Structured error fields (see :func:`_structured_error_texts`) are read on + every exit code: they are how a backend that swallows its own failure and + exits 0 -- codex on a rejected refresh, measured on codex-cli 0.130.0 -- + still says what went wrong. - On a **non-zero** exit the raw streams are read too. The invocation has - already failed, so the only question left is which kind of failure it was, - and CLIs routinely report an unusable credential on stderr without ever - emitting a structured event. + On a **non-zero** exit stderr is read as well: CLIs routinely report an + unusable credential there without emitting a structured event. The raw + stdout stream is never scanned on any exit code; for the JSONL backends it + is the whole transcript, tool output and agent prose included. """ - for text in _errored_envelope_texts(parsed or {}): + for text in _structured_error_texts(backend, parsed, result.stdout or ""): marker = credential_failure_marker(text) if marker is not None: - return marker, "structured result envelope (is_error)" + return marker, "structured error event" if result.returncode == 0: return None - for where, text in (("stderr", result.stderr), ("stdout", result.stdout)): - marker = credential_failure_marker(text or "") - if marker is not None: - return marker, where + marker = credential_failure_marker(result.stderr or "") + if marker is not None: + return marker, "stderr" return None @@ -805,6 +896,10 @@ def _credential_refresh_error( BACKEND_STDERR_ARTIFACT_NAME = ".helix_backend_stderr.txt" BACKEND_TRANSCRIPT_ARTIFACT_DIR = ".helix_artifacts/backend_transcripts" +#: Where a sandboxed opencode candidate keeps its SQLite database: under the +#: per-candidate workspace copy, which the container sees at ``/workspace``. +OPENCODE_SANDBOX_DB_PATH = f"/workspace/{OPENCODE_STATE_DIR_NAME}/opencode/opencode.db" + def _prompt_file_instruction(prompt_artifact_name: str) -> str: return ( @@ -835,12 +930,17 @@ def _ignore_helix_artifacts(worktree_path: Path) -> None: BACKEND_RESULT_ARTIFACT_NAME, BACKEND_STDOUT_ARTIFACT_NAME, BACKEND_STDERR_ARTIFACT_NAME, + # The first attempt's copies after a lost refresh race (see + # ``invoke_with_refresh_race_retry``). + attempt_artifact_name(BACKEND_RESULT_ARTIFACT_NAME, 1), + attempt_artifact_name(BACKEND_STDOUT_ARTIFACT_NAME, 1), + attempt_artifact_name(BACKEND_STDERR_ARTIFACT_NAME, 1), ".helix_artifacts/", "helix_batch.json", # Per-candidate OpenCode SQLite state (OPENCODE_DB isolation). # Each parallel opencode worker gets a fresh database here; keeps # the candidate git tree free of opencode's session transcripts. - ".helix_opencode_state/", + f"{OPENCODE_STATE_DIR_NAME}/", ] existing = gitignore.read_text() if gitignore.exists() else "" to_append = [p for p in patterns if p not in existing] @@ -1686,8 +1786,6 @@ def _copy_local_claude_transcript( } try: dst.parent.mkdir(parents=True, exist_ok=True) - import shutil - shutil.copy2(src, dst) except OSError as exc: return { @@ -1839,6 +1937,190 @@ def _combine_usage( return total + +# --------------------------------------------------------------------------- +# One-shot retry after a lost refresh race +# --------------------------------------------------------------------------- + + +def attempt_artifact_name(name: str, attempt: int) -> str: + """Name under which an earlier attempt's artifact is kept. + + ``.helix_backend_result.json`` -> ``.helix_backend_result.attempt1.json``; + a name without an extension gets the suffix appended. + """ + stem, dot, ext = name.rpartition(".") + if not stem: + return f"{name}.attempt{attempt}" + return f"{stem}.attempt{attempt}{dot}{ext}" + + +_ATTEMPT_ARTIFACT_NAMES: tuple[str, ...] = ( + BACKEND_RESULT_ARTIFACT_NAME, + BACKEND_STDOUT_ARTIFACT_NAME, + BACKEND_STDERR_ARTIFACT_NAME, + BACKEND_TRANSCRIPT_ARTIFACT_DIR, +) + + +def _stash_attempt_artifacts(worktree_path: str, stash_dir: Path) -> None: + """Copy the invocation artifacts of *worktree_path* into *stash_dir*.""" + wt = Path(worktree_path) + for name in _ATTEMPT_ARTIFACT_NAMES: + src = wt / name + dst = stash_dir / name + try: + if src.is_dir(): + shutil.copytree(src, dst) + elif src.is_file(): + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src, dst) + except OSError as exc: + logger.debug("could not stash %s from %s: %s", name, worktree_path, exc) + + +def _restore_attempt_artifacts( + worktree_path: str, stash_dir: Path, *, attempt: int +) -> None: + """Write stashed artifacts into *worktree_path* under their attempt names.""" + wt = Path(worktree_path) + for name in _ATTEMPT_ARTIFACT_NAMES: + src = stash_dir / name + dst = wt / attempt_artifact_name(name, attempt) + try: + if src.is_dir(): + shutil.copytree(src, dst, dirs_exist_ok=True) + elif src.is_file(): + dst.parent.mkdir(parents=True, exist_ok=True) + shutil.copy2(src, dst) + except OSError as exc: + logger.debug("could not restore %s into %s: %s", name, worktree_path, exc) + + +def _note_retry_in_result_artifact( + worktree_path: str, + *, + first_usage: UsageStats | None, + combined_usage: UsageStats, +) -> None: + """Make the final attempt's result artifact account for both attempts. + + The artifact on disk describes the retry only; the caller is charged the + sum. Record the sum and point at the first attempt's artifact so the + file stays a faithful account of what the invocation spent. + """ + path = Path(worktree_path) / BACKEND_RESULT_ARTIFACT_NAME + try: + payload = json.loads(path.read_text()) + except (OSError, ValueError): + return + if not isinstance(payload, dict): + return + payload["attempts"] = 2 + payload["retry_of"] = attempt_artifact_name(BACKEND_RESULT_ARTIFACT_NAME, 1) + payload["retry_reason"] = "lost refresh race on the shared credential" + payload["usage_first_attempt"] = ( + first_usage.to_dict() if first_usage is not None else None + ) + payload["usage_combined"] = combined_usage.to_dict() + try: + path.write_text(json.dumps(payload, indent=2)) + except OSError as exc: + logger.debug("could not annotate %s: %s", path, exc) + + +def invoke_with_refresh_race_retry( + child: Candidate, + *, + backend_name: str, + invoke: Callable[[Candidate, bool], tuple[dict[str, Any], UsageStats]], + fresh_child: Callable[[], Candidate], + remove_child: Callable[[Candidate], None], + on_child_replaced: Callable[[Candidate], None], + record_usage: Callable[[UsageStats], None] | None = None, + on_refresh_race_recovered: Callable[[str], None] | None = None, +) -> tuple[Candidate, UsageStats]: + """Run *invoke* on *child*, retrying once on a lost refresh race. + + A transient :class:`CredentialRefreshError` means another candidate + refreshed the shared login first and the token this invocation held was + already spent; the winner has stored a working credential, so a second + invocation against it is the right response. The retry must not run on + the tree the first attempt was editing: with the sandbox, ``sync_back`` + copies the first attempt's partial edits into the worktree regardless of + exit code, and the retry's workspace copy would commit them as its + synthetic baseline, hiding them from ``git diff``. So the retry starts + from a *fresh* worktree -- ``remove_child`` then ``fresh_child`` -- and + the first attempt's artifacts are kept beside the retry's under + ``attempt1`` names (:func:`attempt_artifact_name`). + + ``on_child_replaced`` is called with the fresh worktree the moment it + exists, so the caller's cleanup handlers always address the live one. + + Usage accounting: the first attempt's spend rides along. On a successful + retry the returned usage is the sum and the result artifact is annotated; + when the retry raises a :class:`HelixError` the sum is attached to it; + when it raises anything else (a sandbox ``TimeoutExpired``, an + ``OSError``) the first attempt's spend is handed to ``record_usage`` + directly before the exception propagates, since nothing else will carry it. + + A recovered race is reported through ``on_refresh_race_recovered`` so it + reaches the operator (the end-of-run summary), not only the log file. + A second loss in a row is raised as-is; there is no loop. + """ + try: + _, usage = invoke(child, False) + return child, usage + except CredentialRefreshError as exc: + if not exc.transient: + raise + first_error = exc + first_usage = exc.usage + + logger.warning( + "%s lost a refresh race on the shared credential; retrying once from " + "a fresh worktree against the refreshed credential (%s).", + backend_name, + first_error, + ) + with tempfile.TemporaryDirectory(prefix="helix-attempt1-") as stash: + _stash_attempt_artifacts(child.worktree_path, Path(stash)) + try: + remove_child(child) + except Exception as exc: # noqa: BLE001 - re-clone below reports the real problem + logger.debug("could not remove %s before retry: %s", child.worktree_path, exc) + child = fresh_child() + on_child_replaced(child) + _restore_attempt_artifacts(child.worktree_path, Path(stash), attempt=1) + + try: + _, usage = invoke(child, True) + except HelixError as retry_exc: + retry_exc.usage = _combine_usage(first_usage, retry_exc.usage) + raise + except Exception: + if record_usage is not None and first_usage is not None: + record_usage(first_usage) + raise + + combined = _combine_usage(first_usage, usage) + assert combined is not None # ``usage`` is never None + _note_retry_in_result_artifact( + child.worktree_path, first_usage=first_usage, combined_usage=combined + ) + message = ( + f"{child.id} recovered after a lost refresh race on the shared " + f"{backend_name} credential: the first attempt reported " + f"{first_error}; the retry from a fresh worktree succeeded. Both " + "attempts' tokens are charged and the first attempt's output is kept " + f"as {attempt_artifact_name(BACKEND_RESULT_ARTIFACT_NAME, 1)}." + ) + logger.warning("%s", message) + if on_refresh_race_recovered is not None: + on_refresh_race_recovered(message) + return child, combined + + def invoke_claude_code( worktree_path: str, prompt: str, @@ -1847,6 +2129,8 @@ def invoke_claude_code( fixed_env: dict[str, str] | None = None, sandbox: SandboxConfig | None = None, prompt_artifact_name: str = MUTATION_PROMPT_ARTIFACT_NAME, + *, + retried: bool = False, ) -> tuple[dict[str, Any], UsageStats]: """Invoke the configured backend CLI in *worktree_path*. @@ -1864,6 +2148,12 @@ def invoke_claude_code( fixed_env: Optional mapping of explicit env var values to inject after passthrough values. + retried: + True when this is the one-shot retry after a lost refresh race (see + :func:`invoke_with_refresh_race_retry`). Only changes what a second + credential failure says to the operator; this function itself never + retries, because a retry has to start from a fresh worktree and only + the caller owns the worktree. Returns ------- @@ -1876,14 +2166,17 @@ def invoke_claude_code( On non-zero return code or JSON decode failure. All errors include the full command, full stdout, full stderr (never truncated), exit code, and working directory. + CredentialRefreshError + When the backend reported, in its own structured error fields or on + stderr, that its stored credential could not be used or refreshed. + ``transient`` is set when it merely lost a refresh race. """ if _MUTATOR_OVERRIDE is not None: return _MUTATOR_OVERRIDE(worktree_path, prompt, config) backend = config.backend backend_name = backend_display_name(backend) - backend_worktree_path = ( - "/workspace" if sandbox is not None and sandbox.enabled else worktree_path - ) + sandboxed = sandbox is not None and sandbox.enabled + backend_worktree_path = "/workspace" if sandboxed else worktree_path args = _build_backend_args( backend_worktree_path, config, @@ -1894,8 +2187,12 @@ def invoke_claude_code( passthrough_env=passthrough_env, fixed_env=fixed_env ) _add_backend_auth_env(backend_env, backend) - backend_env.update(BACKEND_FRESH_SESSION_ENV.get(backend, {})) - if backend == "opencode" and (sandbox is None or not sandbox.enabled): + # The fresh-session switch applies to sandboxed and unsandboxed runs + # alike. An operator who names the same key in ``[env]`` has made a + # deliberate choice, so their value wins. + for key, value in BACKEND_FRESH_SESSION_ENV.get(backend, {}).items(): + backend_env.setdefault(key, value) + if backend == "opencode": # Per-candidate SQLite isolation for concurrent opencode subprocesses. # # OpenCode stores its session database at: @@ -1919,168 +2216,82 @@ def invoke_claude_code( # /.helix_opencode_state/opencode/opencode.db # Each parallel worker gets its own fresh database; no contention. # - # The sandbox branch is excluded: container isolation already provides - # per-candidate filesystem separation, so the knob would be redundant. - opencode_state_dir = Path(worktree_path) / ".helix_opencode_state" / "opencode" - opencode_state_dir.mkdir(parents=True, exist_ok=True) - backend_env["OPENCODE_DB"] = str(opencode_state_dir / "opencode.db") - - def _attempt(*, retried: bool) -> tuple[dict[str, Any], UsageStats]: - """Run the backend once and classify the outcome.""" - if sandbox is not None and sandbox.enabled: - sandbox_image = resolve_sandbox_image(sandbox, backend) - result = run_sandboxed_command( - args, - cwd=worktree_path, - env=backend_env, - sandbox=sandbox, - scope="agent", - sync_back=True, - image=sandbox_image, - agent_backend=backend, - ) + # The sandbox needs the same knob, not less: every sandboxed agent + # container mounts the one ``helix-auth-opencode`` volume at + # ``/home/node`` read-write with ``HOME=/home/node`` forced, so + # without it every concurrent container opens the same + # ``~/.local/share/opencode/opencode.db``. Container isolation + # separates ``/workspace``, which is why the database goes there: + # the container's cwd is the per-candidate workspace copy, the + # sandbox creates the directory in that copy before the run (see + # ``helix.sandbox.run_sandboxed_commands``), and ``.helix*`` paths + # are excluded from sync-back and gitignored like the unsandboxed + # layout. + if sandboxed: + backend_env["OPENCODE_DB"] = OPENCODE_SANDBOX_DB_PATH else: - result = subprocess.run( - args, - cwd=worktree_path, - capture_output=True, - text=True, - env=backend_env, + opencode_state_dir = ( + Path(worktree_path) / OPENCODE_STATE_DIR_NAME / "opencode" ) + opencode_state_dir.mkdir(parents=True, exist_ok=True) + backend_env["OPENCODE_DB"] = str(opencode_state_dir / "opencode.db") + + if sandboxed: + assert sandbox is not None + sandbox_image = resolve_sandbox_image(sandbox, backend) + result = run_sandboxed_command( + args, + cwd=worktree_path, + env=backend_env, + sandbox=sandbox, + scope="agent", + sync_back=True, + image=sandbox_image, + agent_backend=backend, + ) + else: + result = subprocess.run( + args, + cwd=worktree_path, + capture_output=True, + text=True, + env=backend_env, + ) - # Recover the usage record from the raw stream FIRST, with a parse - # that cannot raise. Everything below this line can fail -- and - # when it does, the tokens have still been spent. Charging must - # not be conditional on the candidate being usable, so every error - # raised from here carries this on ``HelixError.usage`` for the - # caller to charge. - spent_usage = _salvage_backend_usage(backend, result) - - parsed: dict[str, Any] | None = None - try: - if result.returncode == 0: - parsed = _parse_backend_output( - backend, - result, - cmd_str=cmd_str, - worktree_path=worktree_path, - ) - usage = _normalise_usage_stats(parsed) - # A backend can report an unusable credential and still exit 0 -- - # measured on codex-cli 0.130.0, whose refresh failure is swallowed - # entirely (exit 0, empty stderr, even at RUST_LOG=info). The - # envelope's own ``is_error`` flag is the only signal left on this - # path, so read it here rather than letting the failure pass as a - # successful-but-useless mutation. - evidence = _credential_failure_evidence(parsed, result) - if evidence is not None: - marker, where = evidence - logger.error( - "Credential failure detected for %s in %s: matched %r", - backend_name, - where, - marker, - ) - raise _credential_refresh_error( - backend=backend, - backend_name=backend_name, - marker=marker, - where=where, - cmd_str=cmd_str, - worktree_path=worktree_path, - result=result, - retried=retried, - ) - if backend == "claude": - error_text = str(parsed.get("error", "")) - if _looks_like_rate_limit(error_text): - logger.error( - "Rate limit detected in JSON response: %s", error_text[:200] - ) - raise RateLimitError( - f"{backend_name} returned a rate/usage limit error in JSON response", - operation=f"{backend_name} invocation", - phase="JSON parsing", - command=cmd_str, - cwd=str(worktree_path), - stdout=result.stdout, - stderr=result.stderr, - exit_code=result.returncode, - suggestion=( - f"{backend_name} reported a rate limit. " - "Retry after backoff or check your API quota." - ), - ) - return parsed, usage - - # Classify a credential failure ahead of the rate-limit and generic - # paths. The markers are disjoint from the rate-limit keywords, and - # "the login is unusable" is a strictly more actionable verdict than - # "the backend exited non-zero". - evidence = _credential_failure_evidence(parsed, result) - if evidence is not None: - marker, where = evidence - logger.error( - "Credential failure detected for %s (exit %d) in %s: matched %r", - backend_name, - result.returncode, - where, - marker, - ) - raise _credential_refresh_error( - backend=backend, - backend_name=backend_name, - marker=marker, - where=where, - cmd_str=cmd_str, - worktree_path=worktree_path, - result=result, - retried=retried, - ) - - rate_limit_source = result.stderr or result.stdout - if _looks_like_rate_limit(rate_limit_source): - logger.error( - "Rate limit detected in subprocess exit for %s (code %d): %s", - backend_name, - result.returncode, - rate_limit_source[:200], - ) - raise RateLimitError( - f"{backend_name} hit a rate/usage limit (exit code {result.returncode})", - operation=f"{backend_name} invocation", - phase="subprocess exit", - command=cmd_str, - cwd=str(worktree_path), - stdout=result.stdout, - stderr=result.stderr, - exit_code=result.returncode, - suggestion=( - f"{backend_name} reported a rate limit. " - "Retry after backoff or check your quota." - ), - ) - - # Claude's max-turns exhaustion is intentionally treated as partial - # success because the subprocess may have already produced useful edits. - if backend == "claude": - try: - parsed = _parse_backend_output( - backend, - result, - cmd_str=cmd_str, - worktree_path=worktree_path, - ) - usage = _normalise_usage_stats(parsed) - if parsed.get("subtype") == "error_max_turns": - logger.warning( - "Claude Code reached max_turns limit (%s turns) — treating as partial success.", - parsed.get("num_turns", "?"), - ) - return parsed, usage - except MutationError: - parsed = None + # Recover the usage record from the raw stream FIRST, with a parse + # that cannot raise. Everything below this line can fail -- and + # when it does, the tokens have still been spent. Charging must + # not be conditional on the candidate being usable, so every error + # raised from here carries this on ``HelixError.usage`` for the + # caller to charge. + spent_usage = _salvage_backend_usage(backend, result) + + def _raise_if_credential_failure(parsed: dict[str, Any] | None) -> None: + evidence = _credential_failure_evidence(backend, parsed, result) + if evidence is None: + return + marker, where = evidence + logger.error( + "Credential failure detected for %s (exit %d) in %s: matched %r", + backend_name, + result.returncode, + where, + marker, + ) + raise _credential_refresh_error( + backend=backend, + backend_name=backend_name, + marker=marker, + where=where, + cmd_str=cmd_str, + worktree_path=worktree_path, + result=result, + retried=retried, + ) + parsed: dict[str, Any] | None = None + try: + if result.returncode == 0: parsed = _parse_backend_output( backend, result, @@ -2088,9 +2299,73 @@ def _attempt(*, retried: bool) -> tuple[dict[str, Any], UsageStats]: worktree_path=worktree_path, ) usage = _normalise_usage_stats(parsed) + # A backend can report an unusable credential and still exit 0 -- + # measured on codex-cli 0.130.0, whose refresh failure is swallowed + # entirely (exit 0, empty stderr, even at RUST_LOG=info). Its + # own structured error event is the only signal left on this + # path, so read it here rather than letting the failure pass as + # a successful-but-useless mutation. + _raise_if_credential_failure(parsed) + if backend == "claude": + error_text = str(parsed.get("error", "")) + if _looks_like_rate_limit(error_text): + logger.error( + "Rate limit detected in JSON response: %s", error_text[:200] + ) + raise RateLimitError( + f"{backend_name} returned a rate/usage limit error in JSON response", + operation=f"{backend_name} invocation", + phase="JSON parsing", + command=cmd_str, + cwd=str(worktree_path), + stdout=result.stdout, + stderr=result.stderr, + exit_code=result.returncode, + suggestion=( + f"{backend_name} reported a rate limit. " + "Retry after backoff or check your API quota." + ), + ) + return parsed, usage - raise MutationError( - f"{backend_name} exited with code {result.returncode}", + # Non-zero exit. Claude's max-turns exhaustion is partial success + # and is decided FIRST: the subprocess may have already produced + # useful edits, and the envelope's ``result`` is the assistant's + # own prose, which no later classifier may read as evidence. + if backend == "claude": + try: + parsed = _parse_backend_output( + backend, + result, + cmd_str=cmd_str, + worktree_path=worktree_path, + ) + except MutationError: + parsed = None + else: + if parsed.get("subtype") == "error_max_turns": + logger.warning( + "Claude Code reached max_turns limit (%s turns) — treating as partial success.", + parsed.get("num_turns", "?"), + ) + return parsed, _normalise_usage_stats(parsed) + + # Then a credential failure, from structured error fields and stderr + # only. The markers are disjoint from the rate-limit keywords, and + # "the login is unusable" is a strictly more actionable verdict than + # "the backend exited non-zero". + _raise_if_credential_failure(parsed) + + rate_limit_source = result.stderr or result.stdout + if _looks_like_rate_limit(rate_limit_source): + logger.error( + "Rate limit detected in subprocess exit for %s (code %d): %s", + backend_name, + result.returncode, + rate_limit_source[:200], + ) + raise RateLimitError( + f"{backend_name} hit a rate/usage limit (exit code {result.returncode})", operation=f"{backend_name} invocation", phase="subprocess exit", command=cmd_str, @@ -2098,50 +2373,46 @@ def _attempt(*, retried: bool) -> tuple[dict[str, Any], UsageStats]: stdout=result.stdout, stderr=result.stderr, exit_code=result.returncode, - suggestion="Check stderr for rate limits, permission errors, or model availability.", - ) - except HelixError as exc: - # Attach only when the raiser did not already supply a more - # precise record; never overwrite one. - if exc.usage is None: - exc.usage = spent_usage - raise - finally: - _write_backend_artifacts( - worktree_path, - backend=backend, - command=cmd_str, - result=result, - parsed=parsed, - sandbox=sandbox, - fallback_usage=spent_usage, + suggestion=( + f"{backend_name} reported a rate limit. " + "Retry after backoff or check your quota." + ), ) - try: - return _attempt(retried=False) - except CredentialRefreshError as exc: - if not exc.transient: - raise - # The lost attempt still spent tokens (``exc.usage`` was attached - # above); fold them into whatever the retry reports so the retry - # does not hide the first attempt's spend from the budget. - first_usage = exc.usage - # Lost a refresh race: another candidate has already stored the - # refreshed credential in the shared volume, so a second invocation - # starts from a working login. One retry; a second loss in a row is - # reported as-is rather than looping. - logger.warning( - "%s lost a refresh race on the shared credential; retrying the " - "invocation once against the refreshed credential (%s).", - backend_name, - exc, + parsed = _parse_backend_output( + backend, + result, + cmd_str=cmd_str, + worktree_path=worktree_path, ) - try: - parsed, usage = _attempt(retried=True) - except HelixError as retry_exc: - retry_exc.usage = _combine_usage(first_usage, retry_exc.usage) + + raise MutationError( + f"{backend_name} exited with code {result.returncode}", + operation=f"{backend_name} invocation", + phase="subprocess exit", + command=cmd_str, + cwd=str(worktree_path), + stdout=result.stdout, + stderr=result.stderr, + exit_code=result.returncode, + suggestion="Check stderr for rate limits, permission errors, or model availability.", + ) + except HelixError as exc: + # Attach only when the raiser did not already supply a more + # precise record; never overwrite one. + if exc.usage is None: + exc.usage = spent_usage raise - return parsed, _combine_usage(first_usage, usage) or usage + finally: + _write_backend_artifacts( + worktree_path, + backend=backend, + command=cmd_str, + result=result, + parsed=parsed, + sandbox=sandbox, + fallback_usage=spent_usage, + ) # --------------------------------------------------------------------------- @@ -2158,6 +2429,7 @@ def mutate( background: str | None = None, prepare_worktree: Callable[[Candidate], None] | None = None, record_usage: Callable[[UsageStats], None] | None = None, + on_refresh_race_recovered: Callable[[str], None] | None = None, ) -> Candidate | None: """Mutate *parent* using the configured backend and return the new candidate. @@ -2184,16 +2456,25 @@ def mutate( are spent either way, so this is how a caller charges the budget for an attempt that ends in ``None``. Not called when no backend invocation happened (e.g. the worktree clone raised). + on_refresh_race_recovered: + Optional sink told, in operator-facing words, when the invocation lost + a refresh race on the shared credential and succeeded on its one retry + from a fresh worktree (see :func:`invoke_with_refresh_race_retry`). Returns ------- Candidate | None The new candidate on success, or ``None`` if mutation failed. """ - child = clone_candidate(parent, new_id, base_dir) - child.operation = "mutate" - if prepare_worktree is not None: - prepare_worktree(child) + + def _fresh_child() -> Candidate: + fresh = clone_candidate(parent, new_id, base_dir) + fresh.operation = "mutate" + if prepare_worktree is not None: + prepare_worktree(fresh) + return fresh + + child = _fresh_child() prompt = build_mutation_prompt( config.objective, @@ -2208,18 +2489,47 @@ def mutate( # per-worktree ``.gitignore`` entry (see ``_ignore_helix_artifacts``) # keep it out of the candidate git tree — otherwise it'd leak into # every subsequent mutation's diff and the mutator would see its own - # prior prompt file as part of the codebase. - prompt_artifact_name = _write_mutation_prompt_artifact(child.worktree_path, prompt) - - try: - _, usage = invoke_claude_code( - child.worktree_path, + # prior prompt file as part of the codebase. Written here, before the + # first invocation, so a collision is raised as-is; the retry path + # rewrites it into its fresh worktree (the write is idempotent). + _write_mutation_prompt_artifact(child.worktree_path, prompt) + + def _invoke( + target: Candidate, retried: bool + ) -> tuple[dict[str, Any], UsageStats]: + return invoke_claude_code( + target.worktree_path, prompt, config.agent, passthrough_env=config.passthrough_env, fixed_env=config.env, sandbox=config.sandbox, - prompt_artifact_name=prompt_artifact_name, + prompt_artifact_name=_write_mutation_prompt_artifact( + target.worktree_path, prompt + ), + retried=retried, + ) + + def _replace(fresh: Candidate) -> None: + nonlocal child + child = fresh + + def _discard_child() -> None: + try: + remove_worktree(child) + except Exception: + pass + + try: + child, usage = invoke_with_refresh_race_retry( + child, + backend_name=backend_display_name(config.agent.backend), + invoke=_invoke, + fresh_child=_fresh_child, + remove_child=remove_worktree, + on_child_replaced=_replace, + record_usage=record_usage, + on_refresh_race_recovered=on_refresh_race_recovered, ) child.usage = usage if record_usage is not None: @@ -2231,10 +2541,7 @@ def mutate( record_usage(exc.usage) exc.operation = f"mutate {new_id} (parent: {parent.id})" print_helix_error(exc) - try: - remove_worktree(child) - except Exception: - pass + _discard_child() return None except RateLimitError as exc: # Rate limit — clean up orphaned worktree, then re-raise so the parallel @@ -2243,20 +2550,25 @@ def mutate( # before the limit hit, so the same handoff applies. if record_usage is not None and exc.usage is not None: record_usage(exc.usage) - try: - remove_worktree(child) - except Exception: - pass + _discard_child() raise - except CredentialRefreshError: - # The stored login, not this candidate, is what failed. Clean up the - # orphaned worktree and re-raise so evolution.py can count it and name - # it as a credential failure instead of filing it under "the agent - # wrote bad code". - try: - remove_worktree(child) - except Exception: - pass + except CredentialRefreshError as exc: + # The stored login, not this candidate, is what failed. The tokens + # spent before the credential gave out are still spent, so hand them + # to the sink like the other paths do; then clean up the orphaned + # worktree and re-raise so evolution.py can count it and name it as a + # credential failure instead of filing it under "the agent wrote bad + # code". + if record_usage is not None and exc.usage is not None: + record_usage(exc.usage) + _discard_child() + raise + except Exception: + # Anything else (a sandbox ``TimeoutExpired``, an ``OSError`` from + # the launch) carries no usage of its own; whatever an earlier + # attempt spent has already been handed to the sink. Do not leak + # the worktree on the way out. + _discard_child() raise # NOTE: snapshot_candidate() is intentionally NOT called here. diff --git a/tests/unit/test_credential_failure_classification.py b/tests/unit/test_credential_failure_classification.py index 5bd1ba2a..926290a2 100644 --- a/tests/unit/test_credential_failure_classification.py +++ b/tests/unit/test_credential_failure_classification.py @@ -21,7 +21,8 @@ import pytest -from helix.config import AgentConfig +from helix.config import AgentConfig, EvaluatorConfig, HelixConfig +from helix.display import UsageStats from helix.exceptions import ( CredentialRefreshError, HelixError, @@ -32,7 +33,10 @@ credential_failure_is_transient, credential_failure_marker, invoke_claude_code, + mutate, ) +from helix.population import Candidate +from tests.unit.test_mutator import make_eval_result # type: ignore[import-untyped] # Codex CLI 0.130.0 -- the four suffixes it appends to one prefix, plus the @@ -170,6 +174,55 @@ def _patch_backend( mocker.patch("helix.mutator.subprocess.run", return_value=result) +def _jsonl(*events: dict[str, Any]) -> str: + return "\n".join(json.dumps(event) for event in events) + + +# The real shapes, read from the CLIs' own event definitions: +# codex ``ThreadErrorEvent`` / ``TurnFailedEvent`` in +# codex-rs/exec/src/exec_events.rs -- no ``is_error`` key exists +# anywhere in ``codex exec --json`` output. +# opencode ``emit("error", { error: props.error })`` in cli/cmd/run.ts, +# where ``props.error`` is ``{ name, data: { message } }``. +def _codex_error_event(message: str) -> dict[str, Any]: + return {"type": "error", "message": message} + + +def _codex_turn_failed(message: str) -> dict[str, Any]: + return {"type": "turn.failed", "error": {"message": message}} + + +def _opencode_error_event(message: str) -> dict[str, Any]: + return { + "type": "error", + "timestamp": 1757500000000, + "sessionID": "ses_x", + "error": {"name": "UnknownError", "data": {"message": message}}, + } + + +def _codex_command_output(output: str) -> dict[str, Any]: + """A completed ``command_execution`` item: the candidate's own tool output.""" + return { + "type": "item.completed", + "item": { + "id": "item_1", + "type": "command_execution", + "command": "pytest -q", + "aggregated_output": output, + "exit_code": 1, + "status": "completed", + }, + } + + +def _codex_agent_message(text: str) -> dict[str, Any]: + return { + "type": "item.completed", + "item": {"id": "item_2", "type": "agent_message", "text": text}, + } + + class TestInvocationClassification: def test_non_zero_exit_with_cli_wording_on_stderr( self, mocker: Any, tmp_path: Path @@ -186,22 +239,20 @@ def test_non_zero_exit_with_cli_wording_on_stderr( assert "credential" in err.suggestion.lower() assert "helix sandbox login codex" in err.suggestion - def test_zero_exit_is_error_envelope_is_read( + def test_zero_exit_codex_error_event_is_read( self, mocker: Any, tmp_path: Path ) -> None: """Codex swallows its own refresh failure: exit 0, empty stderr. Measured on codex-cli 0.130.0 against a synthetic credential whose - refresh was rejected -- the process exits 0 and prints nothing, even at - RUST_LOG=info. The envelope's ``is_error`` flag is the only signal - left, so it has to be read. + refresh was rejected -- the process exits 0 and prints nothing on + stderr, even at RUST_LOG=info. What it does emit is its own + ``{"type": "error", "message": ...}`` event, so that is what has to + be read. """ - stream = "\n".join( - [ - json.dumps({"type": "thread.started"}), - json.dumps({"type": "error", "is_error": True, - "message": CODEX_ALREADY_USED}), - ] + stream = _jsonl( + {"type": "thread.started", "thread_id": "t1"}, + _codex_error_event(CODEX_ALREADY_USED), ) _patch_backend(mocker, returncode=0, stdout=stream) with pytest.raises(CredentialRefreshError) as exc: @@ -209,9 +260,56 @@ def test_zero_exit_is_error_envelope_is_read( str(tmp_path), "p", AgentConfig(backend="codex") ) assert exc.value.exit_code == 0 - assert "is_error" in str(exc.value) + assert exc.value.transient is True + assert "structured error event" in str(exc.value) - def test_claude_top_level_envelope_is_read( + def test_zero_exit_codex_turn_failed_is_read( + self, mocker: Any, tmp_path: Path + ) -> None: + stream = _jsonl( + {"type": "thread.started", "thread_id": "t1"}, + {"type": "turn.started"}, + _codex_turn_failed(CODEX_EXPIRED), + ) + _patch_backend(mocker, returncode=0, stdout=stream) + with pytest.raises(CredentialRefreshError) as exc: + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + assert exc.value.transient is False + + def test_zero_exit_opencode_error_event_is_read( + self, mocker: Any, tmp_path: Path + ) -> None: + stream = _jsonl( + {"type": "step_start", "sessionID": "ses_x"}, + _opencode_error_event(OPENCODE_REFRESH_FAILED), + ) + _patch_backend(mocker, returncode=0, stdout=stream) + with pytest.raises(CredentialRefreshError): + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="opencode") + ) + + def test_claude_errors_list_is_read( + self, mocker: Any, tmp_path: Path + ) -> None: + """Claude Code's error envelope carries an ``errors`` list, not a string.""" + envelope = json.dumps( + { + "type": "result", + "subtype": "error_during_execution", + "is_error": True, + "errors": [CLAUDE_OAUTH_REFRESH], + } + ) + _patch_backend(mocker, returncode=0, stdout=envelope) + with pytest.raises(CredentialRefreshError): + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="claude") + ) + + def test_claude_errored_result_is_read( self, mocker: Any, tmp_path: Path ) -> None: envelope = json.dumps( @@ -228,22 +326,53 @@ def test_claude_top_level_envelope_is_read( str(tmp_path), "p", AgentConfig(backend="claude") ) - def test_is_error_alone_does_not_classify( + def test_claude_successful_result_prose_is_not_read( self, mocker: Any, tmp_path: Path ) -> None: - """An ``is_error`` tool result is usually the agent's own failing - command. That is an ordinary code failure and must stay one.""" - stream = "\n".join( - [ - json.dumps( - { - "type": "tool_result", - "is_error": True, - "content": "pytest exited 1: 2 failed, 9 passed", - } - ), - json.dumps({"type": "turn.completed"}), - ] + """On a successful turn ``result`` is the assistant's own words.""" + envelope = json.dumps( + { + "type": "result", + "subtype": "success", + "is_error": False, + "result": f"I fixed the handler that raised '{CLAUDE_INVALID_KEY}'.", + } + ) + _patch_backend(mocker, returncode=0, stdout=envelope) + parsed, _usage = invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="claude") + ) + assert parsed["subtype"] == "success" + + def test_tool_result_is_not_evidence( + self, mocker: Any, tmp_path: Path + ) -> None: + """A failing tool call is the agent's own failing command -- an + ordinary code failure -- whatever flag the backend puts on it.""" + stream = _jsonl( + { + "type": "tool_result", + "is_error": True, + "content": f"pytest exited 1: {CODEX_ALREADY_USED}", + }, + {"type": "turn.completed"}, + ) + _patch_backend(mocker, returncode=0, stdout=stream) + parsed, _usage = invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + assert parsed["events"] + + def test_agent_prose_with_marker_on_zero_exit_is_not_classified( + self, mocker: Any, tmp_path: Path + ) -> None: + """The agent quoting the CLI's own wording is still prose.""" + stream = _jsonl( + {"type": "thread.started", "thread_id": "t1"}, + _codex_agent_message( + f"The old client printed '{CODEX_ALREADY_USED}'; I removed it." + ), + {"type": "turn.completed", "usage": {"input_tokens": 1}}, ) _patch_backend(mocker, returncode=0, stdout=stream) parsed, _usage = invoke_claude_code( @@ -256,19 +385,12 @@ def test_candidate_editing_oauth_code_is_not_classified( ) -> None: """A candidate whose own work is about refresh tokens must not be able to talk HELIX into declaring the operator's login broken.""" - stream = "\n".join( - [ - json.dumps( - { - "type": "tool_result", - "is_error": True, - "content": ( - "FAILED tests/test_oauth.py::test_reuse - " - "expected the refresh token to be rejected" - ), - } - ) - ] + stream = _jsonl( + _codex_command_output( + "FAILED tests/test_oauth.py::test_reuse - " + "expected the refresh token to be rejected" + ), + {"type": "turn.completed"}, ) _patch_backend(mocker, returncode=0, stdout=stream) parsed, _usage = invoke_claude_code( @@ -276,6 +398,61 @@ def test_candidate_editing_oauth_code_is_not_classified( ) assert parsed["events"] + def test_non_zero_exit_never_scans_tool_output_in_the_transcript( + self, mocker: Any, tmp_path: Path + ) -> None: + """Exit 1 on a 502 after the candidate's tests printed opencode's + refresh wording into a command's output is a generic failure.""" + stream = _jsonl( + {"type": "thread.started", "thread_id": "t1"}, + _codex_command_output(f"E RuntimeError: {OPENCODE_REFRESH_FAILED}"), + ) + _patch_backend( + mocker, returncode=1, stdout=stream, stderr="error: 502 Bad Gateway" + ) + with pytest.raises(MutationError) as exc: + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + assert not isinstance(exc.value, CredentialRefreshError) + + def test_non_zero_exit_never_scans_raw_stdout_prose( + self, mocker: Any, tmp_path: Path + ) -> None: + _patch_backend( + mocker, + returncode=1, + stdout=f"note: {CODEX_EXPIRED}", + stderr="Traceback (most recent call last): SyntaxError", + ) + with pytest.raises(MutationError) as exc: + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + assert not isinstance(exc.value, CredentialRefreshError) + + @pytest.mark.parametrize("is_error", [False, True]) + def test_claude_max_turns_with_auth_wording_in_prose_stays_partial_success( + self, mocker: Any, tmp_path: Path, is_error: bool + ) -> None: + """Pre-existing contract: max-turns exhaustion is partial success + because the edits may be useful. It is decided before any credential + check, so the assistant's prose in ``result`` is never read.""" + envelope = json.dumps( + { + "type": "result", + "subtype": "error_max_turns", + "is_error": is_error, + "num_turns": 30, + "result": f"I hit '{CLAUDE_INVALID_KEY}' in the old test fixture.", + } + ) + _patch_backend(mocker, returncode=1, stdout=envelope) + parsed, _usage = invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="claude") + ) + assert parsed["subtype"] == "error_max_turns" + def test_ordinary_non_zero_exit_stays_a_mutation_error( self, mocker: Any, tmp_path: Path ) -> None: @@ -300,6 +477,34 @@ def test_rate_limit_still_wins_its_own_classification( str(tmp_path), "p", AgentConfig(backend="codex") ) + def test_invoke_itself_never_retries( + self, mocker: Any, tmp_path: Path + ) -> None: + """The retry needs a fresh worktree, which only the caller owns.""" + run = mocker.patch( + "helix.mutator.subprocess.run", + return_value=_completed(1, stderr=CODEX_ALREADY_USED), + ) + with pytest.raises(CredentialRefreshError) as exc: + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex") + ) + assert run.call_count == 1 + assert exc.value.transient is True + assert "retry" not in str(exc.value).lower() + + def test_retried_flag_changes_the_second_loss_wording( + self, mocker: Any, tmp_path: Path + ) -> None: + _patch_backend(mocker, returncode=1, stderr=CODEX_ALREADY_USED) + with pytest.raises(CredentialRefreshError) as exc: + invoke_claude_code( + str(tmp_path), "p", AgentConfig(backend="codex"), retried=True + ) + assert "failed again on retry" in str(exc.value) + assert "helix resume" in exc.value.suggestion + assert "sandbox login" not in exc.value.suggestion + def _completed( returncode: int, stdout: str = "", stderr: str = "" @@ -309,7 +514,104 @@ def _completed( ) -CODEX_SUCCESS_STREAM = json.dumps({"type": "turn.completed"}) +def _codex_stream(*, session: str, input_tokens: int, output_tokens: int) -> str: + return _jsonl( + {"type": "session.started", "session_id": session}, + { + "type": "turn.completed", + "usage": { + "input_tokens": input_tokens, + "output_tokens": output_tokens, + }, + }, + ) + + +CODEX_SUCCESS_STREAM = _codex_stream(session="won", input_tokens=12, output_tokens=8) +CODEX_LOST_STREAM = _codex_stream(session="lost", input_tokens=5, output_tokens=2) + + +def _codex_config() -> HelixConfig: + return HelixConfig( + objective="Improve the code", + evaluator=EvaluatorConfig(command="true"), + agent=AgentConfig(backend="codex"), + ) + + +def _candidate(cid: str, path: Path) -> Candidate: + return Candidate( + id=cid, + worktree_path=str(path), + branch_name=f"helix/{cid}", + generation=0, + parent_id=None, + parent_ids=[], + operation="seed", + ) + + +class _RetryHarness: + """``mutate()`` with a clone that hands out a fresh directory per call. + + Each ``clone_candidate`` call creates ``/clone`` so the test can + see which tree each attempt ran in, what it left behind, and which trees + were removed. + """ + + def __init__(self, tmp_path: Path, mocker: Any) -> None: + self.tmp_path = tmp_path + self.clones: list[Candidate] = [] + self.removed: list[Candidate] = [] + self.run_cwds: list[str] = [] + self.spent: list[UsageStats] = [] + self.recovered: list[str] = [] + self.parent = _candidate("g0-s0", tmp_path / "parent") + + def _clone(parent: Candidate, new_id: str, base_dir: Path) -> Candidate: + path = tmp_path / f"clone{len(self.clones) + 1}" + path.mkdir() + child = _candidate(new_id, path) + self.clones.append(child) + return child + + mocker.patch("helix.mutator.clone_candidate", side_effect=_clone) + mocker.patch( + "helix.mutator.remove_worktree", side_effect=self.removed.append + ) + mocker.patch("helix.mutator.snapshot_candidate") + self.mocker = mocker + + def run_backend(self, outcomes: list[Any]) -> Any: + """Patch ``subprocess.run`` with per-attempt outcomes. + + An outcome is a ``CompletedProcess`` to return or an exception to + raise. Every attempt first drops ``partial.py`` into its cwd, the way + a backend that lost mid-turn leaves half-applied edits behind. + """ + outcomes = list(outcomes) + + def _run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: + cwd = kwargs["cwd"] + self.run_cwds.append(cwd) + (Path(cwd) / "partial.py").write_text("# half-applied edit\n") + outcome = outcomes.pop(0) + if isinstance(outcome, BaseException): + raise outcome + return outcome + + return self.mocker.patch("helix.mutator.subprocess.run", side_effect=_run) + + def mutate(self) -> Candidate | None: + return mutate( + self.parent, + make_eval_result("g0-s0"), + "g1-s0", + _codex_config(), + self.tmp_path, + record_usage=self.spent.append, + on_refresh_race_recovered=self.recovered.append, + ) class TestLostRefreshRaceIsRetried: @@ -318,204 +620,292 @@ class TestLostRefreshRaceIsRetried: When it happens, the *winner* has just written a refreshed credential to the shared volume, so the right response is to invoke again against it, not to drop the slot and tell the operator to redo a login that is fine. + The retry lives in ``mutate()`` / ``merge()`` rather than in + ``invoke_claude_code`` because it must start from a fresh worktree. """ - def test_already_used_is_retried_once_and_the_retry_can_succeed( + def test_retry_runs_in_a_fresh_worktree( self, mocker: Any, tmp_path: Path ) -> None: - run = mocker.patch( - "helix.mutator.subprocess.run", - side_effect=[ - _completed(1, stderr=CODEX_ALREADY_USED), + """Attempt 1's half-applied edits must not be under the retry.""" + h = _RetryHarness(tmp_path, mocker) + run = h.run_backend( + [ + _completed(1, stdout=CODEX_LOST_STREAM, stderr=CODEX_ALREADY_USED), _completed(0, stdout=CODEX_SUCCESS_STREAM), - ], - ) - parsed, _usage = invoke_claude_code( - str(tmp_path), "p", AgentConfig(backend="codex") + ] ) - assert parsed["events"] + + child = h.mutate() + assert run.call_count == 2 + assert len(h.clones) == 2 + assert h.run_cwds == [h.clones[0].worktree_path, h.clones[1].worktree_path] + assert child is h.clones[1] + # The first tree was discarded before the retry, and the retry's tree + # was clean when the backend started in it. + assert h.removed == [h.clones[0]] + retry_tree = Path(h.clones[1].worktree_path) + assert sorted(p.name for p in retry_tree.iterdir() if p.name == "partial.py") == [ + "partial.py" + ], "only the retry's own edit is present" + assert not (retry_tree / ".helix_backend_result.attempt1.json").read_text().count( + "partial" + ) - def test_zero_exit_already_used_envelope_is_retried_too( + def test_both_attempts_artifacts_are_kept( self, mocker: Any, tmp_path: Path ) -> None: - """Codex swallows the failure on exit 0; the envelope path retries as well.""" - stream = "\n".join( + h = _RetryHarness(tmp_path, mocker) + h.run_backend( [ - json.dumps({"type": "thread.started"}), - json.dumps({"type": "error", "is_error": True, - "message": CODEX_ALREADY_USED}), - ] - ) - run = mocker.patch( - "helix.mutator.subprocess.run", - side_effect=[ - _completed(0, stdout=stream), + _completed(1, stdout=CODEX_LOST_STREAM, stderr=CODEX_ALREADY_USED), _completed(0, stdout=CODEX_SUCCESS_STREAM), - ], - ) - parsed, _usage = invoke_claude_code( - str(tmp_path), "p", AgentConfig(backend="codex") + ] ) - assert parsed["events"] - assert run.call_count == 2 - def test_second_loss_is_raised_without_a_relogin_instruction( + child = h.mutate() + assert child is not None + tree = Path(child.worktree_path) + + # Attempt 1, under its own names, with the marker that explains why + # there was a retry. + assert (tree / ".helix_backend_stderr.attempt1.txt").read_text() == CODEX_ALREADY_USED + assert (tree / ".helix_backend_stdout.attempt1.txt").read_text() == CODEX_LOST_STREAM + first = json.loads((tree / ".helix_backend_result.attempt1.json").read_text()) + assert first["returncode"] == 1 + assert first["usage"]["input_tokens"] == 5 + + # The final attempt's artifact accounts for both. + final = json.loads((tree / ".helix_backend_result.json").read_text()) + assert final["returncode"] == 0 + assert final["attempts"] == 2 + assert final["retry_of"] == ".helix_backend_result.attempt1.json" + assert final["usage_first_attempt"]["input_tokens"] == 5 + assert final["usage_combined"]["input_tokens"] == 5 + 12 + assert final["usage_combined"]["output_tokens"] == 2 + 8 + + # And none of it can leak into the candidate's git tree. + gitignore = (tree / ".gitignore").read_text() + for name in ( + ".helix_backend_result.attempt1.json", + ".helix_backend_stdout.attempt1.txt", + ".helix_backend_stderr.attempt1.txt", + ): + assert name in gitignore + + def test_recovered_race_is_reported_and_charged_once( self, mocker: Any, tmp_path: Path ) -> None: - run = mocker.patch( - "helix.mutator.subprocess.run", - side_effect=[ - _completed(1, stderr=CODEX_ALREADY_USED), - _completed(1, stderr=CODEX_ALREADY_USED), - ], - ) - with pytest.raises(CredentialRefreshError) as exc: - invoke_claude_code( - str(tmp_path), "p", AgentConfig(backend="codex") - ) - err = exc.value - assert run.call_count == 2 # exactly one retry, no loop - assert err.transient is True - assert "retry" in str(err).lower() - # The stored credential is the winner's fresh one; do not tell the - # operator to throw it away. - assert "sandbox login" not in err.suggestion - assert "helix resume" in err.suggestion - - @pytest.mark.parametrize("text", [CODEX_EXPIRED, CODEX_REVOKED, CODEX_BARE]) - def test_a_dead_login_is_not_retried( - self, mocker: Any, tmp_path: Path, text: str - ) -> None: - """Retrying an expired or revoked credential only burns a turn.""" - run = mocker.patch( - "helix.mutator.subprocess.run", - side_effect=[_completed(1, stderr=text)], + h = _RetryHarness(tmp_path, mocker) + h.run_backend( + [ + _completed(1, stdout=CODEX_LOST_STREAM, stderr=CODEX_ALREADY_USED), + _completed(0, stdout=CODEX_SUCCESS_STREAM), + ] ) - with pytest.raises(CredentialRefreshError) as exc: - invoke_claude_code( - str(tmp_path), "p", AgentConfig(backend="codex") - ) - assert run.call_count == 1 - assert exc.value.transient is False - assert "helix sandbox login codex" in exc.value.suggestion - - -def _codex_stream(*, session: str, input_tokens: int, output_tokens: int) -> str: - return "\n".join( - [ - json.dumps({"type": "session.started", "session_id": session}), - json.dumps( - { - "type": "turn.completed", - "usage": { - "input_tokens": input_tokens, - "output_tokens": output_tokens, - }, - } - ), - ] - ) - -class TestRetryUsageAccounting: - """A retried invocation spent tokens twice; the caller must see both. - - The lost attempt raises a transient ``CredentialRefreshError`` carrying - the usage salvaged from its output. Whatever the retry then reports -- - a candidate or another error -- has to include that first spend, or the - budget silently under-counts every lost race. - """ - - def test_retry_success_includes_the_lost_attempts_usage( + child = h.mutate() + assert child is not None + + # The caller sees the sum, exactly once, and the child carries it. + assert len(h.spent) == 1 + assert h.spent[0].input_tokens == 5 + 12 + assert h.spent[0].output_tokens == 2 + 8 + assert h.spent[0].session_id == "won" + assert child.usage.input_tokens == 5 + 12 + # The race is visible to whoever keeps the end-of-run summary. + assert len(h.recovered) == 1 + assert "recovered after a lost refresh race" in h.recovered[0] + assert "g1-s0" in h.recovered[0] + + def test_zero_exit_codex_error_event_is_retried_too( self, mocker: Any, tmp_path: Path ) -> None: - run = mocker.patch( - "helix.mutator.subprocess.run", - side_effect=[ - _completed( - 1, - stdout=_codex_stream( - session="lost", input_tokens=5, output_tokens=2 - ), - stderr=CODEX_ALREADY_USED, - ), + """Codex swallows the failure on exit 0; the event path retries as well.""" + h = _RetryHarness(tmp_path, mocker) + run = h.run_backend( + [ _completed( 0, - stdout=_codex_stream( - session="won", input_tokens=12, output_tokens=8 + stdout=_jsonl( + {"type": "thread.started", "thread_id": "t1"}, + _codex_error_event(CODEX_ALREADY_USED), ), ), - ], - ) - parsed, usage = invoke_claude_code( - str(tmp_path), "p", AgentConfig(backend="codex") + _completed(0, stdout=CODEX_SUCCESS_STREAM), + ] ) + assert h.mutate() is h.clones[1] assert run.call_count == 2 - assert parsed["events"] - assert usage.input_tokens == 5 + 12 - assert usage.output_tokens == 2 + 8 - # The session the caller receives output from is the retry's. - assert usage.session_id == "won" - def test_second_loss_carries_the_sum_of_both_attempts( + def test_second_loss_is_raised_with_the_sum_and_no_relogin_instruction( self, mocker: Any, tmp_path: Path ) -> None: - mocker.patch( - "helix.mutator.subprocess.run", - side_effect=[ - _completed( - 1, - stdout=_codex_stream( - session="lost-1", input_tokens=5, output_tokens=2 - ), - stderr=CODEX_ALREADY_USED, - ), + h = _RetryHarness(tmp_path, mocker) + run = h.run_backend( + [ + _completed(1, stdout=CODEX_LOST_STREAM, stderr=CODEX_ALREADY_USED), _completed( 1, - stdout=_codex_stream( - session="lost-2", input_tokens=3, output_tokens=1 - ), + stdout=_codex_stream(session="lost-2", input_tokens=3, output_tokens=1), stderr=CODEX_ALREADY_USED, ), - ], + ] ) with pytest.raises(CredentialRefreshError) as exc: - invoke_claude_code( - str(tmp_path), "p", AgentConfig(backend="codex") - ) - assert exc.value.usage is not None - assert exc.value.usage.input_tokens == 5 + 3 - assert exc.value.usage.output_tokens == 2 + 1 - assert exc.value.usage.session_id == "lost-2" + h.mutate() + err = exc.value + assert run.call_count == 2 # exactly one retry, no loop + assert err.transient is True + assert "failed again on retry" in str(err) + # The stored credential is the winner's fresh one; do not tell the + # operator to throw it away. + assert "sandbox login" not in err.suggestion + assert "helix resume" in err.suggestion + # Both attempts' spend, attached to the error and handed to the sink. + assert err.usage is not None + assert err.usage.input_tokens == 5 + 3 + assert err.usage.output_tokens == 2 + 1 + assert err.usage.session_id == "lost-2" + assert h.spent == [err.usage] + # Neither worktree leaks. + assert h.removed == h.clones + assert h.recovered == [] def test_retry_failing_for_another_reason_still_carries_the_sum( self, mocker: Any, tmp_path: Path ) -> None: """The retry's error class does not matter; the first spend rides along.""" - mocker.patch( - "helix.mutator.subprocess.run", - side_effect=[ - _completed( - 1, - stdout=_codex_stream( - session="lost", input_tokens=5, output_tokens=2 - ), - stderr=CODEX_ALREADY_USED, - ), + h = _RetryHarness(tmp_path, mocker) + h.run_backend( + [ + _completed(1, stdout=CODEX_LOST_STREAM, stderr=CODEX_ALREADY_USED), _completed( 2, - stdout=_codex_stream( - session="crashed", input_tokens=4, output_tokens=0 - ), + stdout=_codex_stream(session="crashed", input_tokens=4, output_tokens=0), stderr="segfault", ), - ], + ] ) - with pytest.raises(MutationError) as exc: - invoke_claude_code( - str(tmp_path), "p", AgentConfig(backend="codex") + assert h.mutate() is None # MutationError -> None, by contract + assert len(h.spent) == 1 + assert h.spent[0].input_tokens == 5 + 4 + assert h.spent[0].output_tokens == 2 + assert h.removed == h.clones + + def test_timeout_on_the_retry_records_the_first_spend_and_cleans_up( + self, mocker: Any, tmp_path: Path + ) -> None: + """A sandbox ``TimeoutExpired`` is not a HelixError and carries no + usage: the first attempt's spend must still reach the sink, and the + retry's worktree must not leak.""" + h = _RetryHarness(tmp_path, mocker) + h.run_backend( + [ + _completed(1, stdout=CODEX_LOST_STREAM, stderr=CODEX_ALREADY_USED), + subprocess.TimeoutExpired(cmd=["codex"], timeout=30), + ] + ) + with pytest.raises(subprocess.TimeoutExpired): + h.mutate() + assert len(h.spent) == 1 + assert h.spent[0].input_tokens == 5 + assert h.spent[0].output_tokens == 2 + assert h.removed == h.clones + + def test_timeout_on_the_first_attempt_cleans_up( + self, mocker: Any, tmp_path: Path + ) -> None: + h = _RetryHarness(tmp_path, mocker) + h.run_backend([subprocess.TimeoutExpired(cmd=["codex"], timeout=30)]) + with pytest.raises(subprocess.TimeoutExpired): + h.mutate() + assert h.spent == [] + assert h.removed == h.clones == h.clones[:1] + + @pytest.mark.parametrize("text", [CODEX_EXPIRED, CODEX_REVOKED, CODEX_BARE]) + def test_a_dead_login_is_not_retried( + self, mocker: Any, tmp_path: Path, text: str + ) -> None: + """Retrying an expired or revoked credential only burns a turn.""" + h = _RetryHarness(tmp_path, mocker) + run = h.run_backend([_completed(1, stdout=CODEX_LOST_STREAM, stderr=text)]) + with pytest.raises(CredentialRefreshError) as exc: + h.mutate() + assert run.call_count == 1 + assert len(h.clones) == 1 + assert exc.value.transient is False + assert "helix sandbox login codex" in exc.value.suggestion + # Item: the credential path records usage like its siblings. + assert len(h.spent) == 1 + assert h.spent[0].input_tokens == 5 + assert h.removed == h.clones + + +class TestRetryOnARealWorktree: + """The same contract against real git worktrees, not a stubbed clone.""" + + def test_retry_starts_from_the_parent_commit( + self, mocker: Any, tmp_path: Path, monkeypatch: pytest.MonkeyPatch + ) -> None: + from helix.worktree import create_seed_worktree + + for key, value in { + "GIT_AUTHOR_NAME": "HELIX Test", + "GIT_AUTHOR_EMAIL": "helix@test.local", + "GIT_COMMITTER_NAME": "HELIX Test", + "GIT_COMMITTER_EMAIL": "helix@test.local", + }.items(): + monkeypatch.setenv(key, value) + project = tmp_path / "project" + project.mkdir() + (project / "main.py").write_text("print('hello')\n") + base_dir = tmp_path / "worktrees" + seed = create_seed_worktree(project, base_dir) + + seen: list[dict[str, Any]] = [] + outcomes = [ + _completed(1, stdout=CODEX_LOST_STREAM, stderr=CODEX_ALREADY_USED), + _completed(0, stdout=CODEX_SUCCESS_STREAM), + ] + + real_run = subprocess.run + + def _run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: + if args[0] != "codex": + # git, driven by the real clone_candidate / remove_worktree + return real_run(args, **kwargs) + cwd = Path(kwargs["cwd"]) + seen.append( + { + "cwd": str(cwd), + "partial_present": (cwd / "partial.py").exists(), + "main": (cwd / "main.py").read_text(), + } ) - assert exc.value.usage is not None - assert exc.value.usage.input_tokens == 5 + 4 - assert exc.value.usage.output_tokens == 2 + (cwd / "partial.py").write_text("# half-applied\n") + (cwd / "main.py").write_text("print('half edited')\n") + return outcomes.pop(0) + + mocker.patch("helix.mutator.subprocess.run", side_effect=_run) + recovered: list[str] = [] + child = mutate( + seed, + make_eval_result(seed.id), + "g1-s0", + _codex_config(), + base_dir, + on_refresh_race_recovered=recovered.append, + ) + + assert child is not None + assert len(seen) == 2 + assert seen[0]["cwd"] == seen[1]["cwd"] == str(base_dir / "g1-s0") + # The retry saw the parent's tree, not attempt 1's half-applied edits. + assert seen[1]["partial_present"] is False + assert seen[1]["main"] == "print('hello')\n" + tree = Path(child.worktree_path) + assert (tree / ".helix_backend_result.attempt1.json").is_file() + assert (tree / ".helix_backend_result.json").is_file() + assert recovered and "recovered after a lost refresh race" in recovered[0] diff --git a/tests/unit/test_merger.py b/tests/unit/test_merger.py index 7a8acd89..a954f5a5 100644 --- a/tests/unit/test_merger.py +++ b/tests/unit/test_merger.py @@ -372,6 +372,65 @@ def test_credential_error_hands_spent_usage_to_the_sink(self, mocker): assert spent == [spent_before_failure] + def test_lost_refresh_race_is_retried_once_from_a_fresh_worktree( + self, mocker, tmp_path: Path + ): + """``merge()`` gets the same one-shot retry as ``mutate()``: a + transient credential failure re-clones the merge worktree and + invokes again; the first attempt's spend rides along.""" + from helix.display import UsageStats + + ca = make_candidate("g0-s0") + cb = make_candidate("g0-s1") + config = make_config() + clones: list = [] + + def _clone(parent, new_id, base_dir): + path = tmp_path / f"clone{len(clones) + 1}" + path.mkdir() + child = make_candidate(new_id) + child.worktree_path = str(path) + clones.append(child) + return child + + mocker.patch("helix.merger.clone_candidate", side_effect=_clone) + mocker.patch("helix.merger.get_diff", return_value="some diff") + lost = CredentialRefreshError( + "lost a refresh race", usage=UsageStats(input_tokens=5, output_tokens=2) + ) + lost.transient = True + invoke = mocker.patch( + "helix.merger.invoke_claude_code", + side_effect=[lost, ({}, UsageStats(input_tokens=12, output_tokens=8))], + ) + removed: list = [] + mocker.patch("helix.merger.remove_worktree", side_effect=removed.append) + mocker.patch("helix.merger.snapshot_candidate") + recovered: list[str] = [] + spent: list[UsageStats] = [] + + result = merge( + ca, + cb, + "g1-m0", + config, + Path("/tmp"), + record_usage=spent.append, + on_refresh_race_recovered=recovered.append, + ) + + assert invoke.call_count == 2 + assert len(clones) == 2 + assert result is clones[1] + assert result.operation == "merge" + assert result.parent_ids == ["g0-s0", "g0-s1"] + # The retry ran in the fresh clone, and the first one was removed. + assert invoke.call_args_list[1].args[0] == clones[1].worktree_path + assert invoke.call_args_list[1].kwargs["retried"] is True + assert removed == [clones[0]] + assert spent[0].input_tokens == 5 + 12 and spent[0].output_tokens == 2 + 8 + assert recovered and "g1-m0" in recovered[0] + def test_snapshot_not_called_by_merge_on_success(self, mocker): """merge() must NOT call snapshot_candidate — the caller owns that step. diff --git a/tests/unit/test_mutator.py b/tests/unit/test_mutator.py index 98ebb3de..fe9dcecd 100644 --- a/tests/unit/test_mutator.py +++ b/tests/unit/test_mutator.py @@ -1918,6 +1918,36 @@ def test_opencode_subprocess_isolation_env_set(self, tmp_path: Path, mocker): "XDG_DATA_HOME must not be set for opencode; it moves auth.json too" ) + def test_sandboxed_opencode_gets_a_per_candidate_database_too( + self, tmp_path: Path, mocker + ): + """Every sandboxed container shares one ``/home/node`` (the + ``helix-auth-opencode`` volume with ``HOME`` forced), so without the + knob all concurrent candidates open the same ``opencode.db``. The + database goes under the per-candidate workspace copy instead.""" + mock_run = mocker.patch("helix.mutator.run_sandboxed_command") + mock_run.return_value = MagicMock( + stdout='{"type":"result","sessionID":"ses_abc"}\n', + stderr="", + returncode=0, + ) + + invoke_claude_code( + str(tmp_path), + "fix the bug", + AgentConfig(backend="opencode"), + sandbox=SandboxConfig(enabled=True, image="img:latest"), + ) + + env = mock_run.call_args[1]["env"] + assert env["OPENCODE_DB"] == ( + "/workspace/.helix_opencode_state/opencode/opencode.db" + ) + assert "XDG_DATA_HOME" not in env + # The host worktree is untouched: the directory is created in the + # workspace copy by the sandbox, not here. + assert not (tmp_path / ".helix_opencode_state").exists() + def test_opencode_subprocess_isolation_unique_per_candidate( self, tmp_path: Path, mocker ): @@ -2107,6 +2137,60 @@ def test_nothing_is_reported_when_the_error_carries_no_usage( assert result is None assert spent == [] + def test_usage_is_reported_when_the_credential_fails( + self, tmp_path: Path, mocker + ): + """A credential failure is not free: the tokens spent before the + login gave out reach the sink like the sibling error paths.""" + from helix.exceptions import CredentialRefreshError + + usage = UsageStats(input_tokens=11, output_tokens=4) + parent = make_candidate("g0-s0") + er = make_eval_result() + config = make_config() + child_path = tmp_path / "g1-s0" + child_path.mkdir() + child = make_candidate("g1-s0", str(child_path)) + mocker.patch("helix.mutator.clone_candidate", return_value=child) + mocker.patch( + "helix.mutator.invoke_claude_code", + side_effect=CredentialRefreshError("login is dead", usage=usage), + ) + mock_remove = mocker.patch("helix.mutator.remove_worktree") + mocker.patch("helix.mutator.snapshot_candidate") + + spent: list[UsageStats] = [] + with pytest.raises(CredentialRefreshError): + mutate( + parent, er, "g1-s0", config, Path("/tmp"), record_usage=spent.append + ) + + assert spent == [usage] + mock_remove.assert_called_once_with(child) + + def test_worktree_is_removed_on_a_non_helix_exception( + self, tmp_path: Path, mocker + ): + import subprocess as _sp + + parent = make_candidate("g0-s0") + er = make_eval_result() + config = make_config() + child_path = tmp_path / "g1-s0" + child_path.mkdir() + child = make_candidate("g1-s0", str(child_path)) + mocker.patch("helix.mutator.clone_candidate", return_value=child) + mocker.patch( + "helix.mutator.invoke_claude_code", + side_effect=_sp.TimeoutExpired(cmd=["codex"], timeout=1), + ) + mock_remove = mocker.patch("helix.mutator.remove_worktree") + + with pytest.raises(_sp.TimeoutExpired): + mutate(parent, er, "g1-s0", config, Path("/tmp")) + + mock_remove.assert_called_once_with(child) + class TestSalvageBackendUsage: """The lenient recovery used before the strict parse can fail.""" From 3469217d6a2cec910c44a47e4b35a2a69c6bf4ac Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Thu, 10 Sep 2026 21:10:13 -0700 Subject: [PATCH 13/16] backends: run auth commands with sh -c, not a login shell sh -lc sources /etc/profile and $HOME/.profile from the shared login volume that every candidate container mounts read-write, so a candidate could plant code that runs in the next warm, status or logout. PATH is pinned with -e by sandbox_auth_docker_args; nothing needs a profile. Applies to the codex warm and the pre-existing agy/claude entries. The codex warm's comment in the same hunk is rewritten for the next commit (exit code says nothing; the catalog cache is written to the volume). A registry test asserting no auth command uses -l lands with the warm tests in the next commit. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0126UoDvKj2BN5SHLH81aqnW --- src/helix/backends.py | 33 ++++++++++++++++++++++++--------- tests/unit/test_sandbox.py | 6 +++--- 2 files changed, 27 insertions(+), 12 deletions(-) diff --git a/src/helix/backends.py b/src/helix/backends.py index 0dfeca52..c90d9afa 100644 --- a/src/helix/backends.py +++ b/src/helix/backends.py @@ -95,6 +95,11 @@ "claude": {"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1"}, } +# Every shell entry below is ``sh -c``, never ``sh -lc``: a login shell sources +# ``/etc/profile`` and ``$HOME/.profile`` from the shared login volume, which +# every candidate container mounts read-write, so ``-l`` would let a candidate +# plant code that runs in the next auth command. PATH is pinned with ``-e`` by +# ``helix.sandbox.sandbox_auth_docker_args``; nothing here needs a profile. BACKEND_AUTH_COMMANDS: dict[str, dict[str, list[str]]] = { "agy": { # No dedicated non-interactive login subcommand; the bare interactive @@ -107,7 +112,7 @@ # confirmed against a completed sign-in. "status": [ "sh", - "-lc", + "-c", 'set -eu; test -s "${HOME:-/home/node}/.gemini/antigravity-cli/antigravity-oauth-token"', ], # Surgical: only remove agy's own state directory. ``~/.gemini`` also @@ -115,7 +120,7 @@ # would destroy state this backend does not own. "logout": [ "sh", - "-lc", + "-c", 'set -eu; rm -rf "${HOME:-/home/node}/.gemini/antigravity-cli"', ], }, @@ -128,7 +133,7 @@ # localised in some CLI versions). "status": [ "sh", - "-lc", + "-c", "set -eu; " "claude auth status --text 2>&1 || true; " 'test -s "${HOME:-/home/node}/.claude/.credentials.json"', @@ -149,18 +154,28 @@ # old. It reads auth.json; it never takes the refresh path, so warming # with it would be a placebo. # - # ``codex debug models`` renders the CLI's built-in model catalog. It - # loads auth through the refreshing path, so it performs the refresh - # this warm exists to perform, and it is free: - # * with a fresh credential it completes with ``--network none``, - # writes nothing to the login volume, and makes no request at all; + # ``codex debug models`` renders the CLI's model catalog. It loads + # auth through the refreshing path, so it performs the refresh this + # warm exists to perform, and it is free: + # * with a fresh credential it completes with ``--network none`` and + # makes no request at all -- the only thing it may write to the + # login volume is its own ``~/.codex/models_cache.json`` (a + # catalog cache with a 5-minute TTL), never the credential; # * with a stale credential its only request is the OAuth token # exchange, which the refreshed credential is then written back # from. No model is invoked and no quota is consumed either way. + # Its exit code says nothing about the refresh: a rejected exchange is + # logged and swallowed and the command still exits 0. That is why + # ``helix.sandbox.warm_backend_credential`` reads ``last_refresh`` + # back from ``auth.json`` and only reports ``warmed`` when the + # credential is verifiably inside codex's refresh interval. There is + # no flag to bypass the catalog cache (``--bundled`` does the + # opposite: it skips the refresh), so a cache younger than 5 minutes + # can short-circuit the refresh; the read-back catches that too. # stdout is discarded because the catalog is ~200 KB and the command is # run for its side effect on the credential, not for its output; # stderr is kept so a failure stays diagnosable. - "warm": ["sh", "-lc", "set -eu; codex debug models >/dev/null"], + "warm": ["sh", "-c", "set -eu; codex debug models >/dev/null"], }, "cursor": { "login": ["cursor-agent", "login"], diff --git a/tests/unit/test_sandbox.py b/tests/unit/test_sandbox.py index 7c36ce47..9f6262f9 100644 --- a/tests/unit/test_sandbox.py +++ b/tests/unit/test_sandbox.py @@ -907,7 +907,7 @@ def test_sandbox_auth_status_command_uses_backend_command(): ) assert "helix-auth-claude:/home/node:rw" in args - assert args[-3:-1] == ["sh", "-lc"] + assert args[-3:-1] == ["sh", "-c"] script = args[-1] assert script.startswith("set -eu; ") assert "claude auth status --text" in script @@ -965,7 +965,7 @@ def test_sandbox_auth_agy_status_uses_credential_file_probe(): ) assert "helix-auth-agy:/home/node:rw" in args - assert args[-3:-1] == ["sh", "-lc"] + assert args[-3:-1] == ["sh", "-c"] script = args[-1] assert ( 'test -s "${HOME:-/home/node}/.gemini/antigravity-cli/antigravity-oauth-token"' @@ -982,7 +982,7 @@ def test_sandbox_auth_agy_logout_only_removes_its_own_state_directory(): action="logout", ) - assert args[-3:-1] == ["sh", "-lc"] + assert args[-3:-1] == ["sh", "-c"] script = args[-1] assert '"${HOME:-/home/node}/.gemini/antigravity-cli"' in script assert script.count(".gemini") == 1 From c5cc0565f87356e2b8bc67c1760a1bbbef4db1c8 Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Thu, 10 Sep 2026 21:10:13 -0700 Subject: [PATCH 14/16] sandbox: verify the codex credential is fresh after the warm; pass operator env to the warm 'codex debug models' logs and swallows a rejected token exchange and exits 0, so the exit code said nothing. warmed=True now means the credential was verified fresh: last_refresh is read from ~/.codex/auth.json before and after the warm through the same single-writer auth container (one sed over one field, --network none; the tokens never leave the container) and judged against codex's refresh rule -- TOKEN_REFRESH_INTERVAL = 8 days in codex-rs login/src/auth/manager.rs (should_refresh_proactively). Advanced, or unchanged but inside the interval, is fresh; unchanged and past the interval is reported as not warmed with a WARNING naming the likely causes (rejected refresh, or a models_cache.json under its 5-minute TTL short-circuiting the refresh -- 'codex debug models' has no cache-bypass flag; --bundled skips the refresh instead) and that candidates will race. The "writes nothing to the login volume" claim is corrected: the catalog cache is written, the credential is not. The warm container now receives the operator's passthrough_env / [env] (proxy, CA), forwarded with -e; sandbox_auth_docker_args and run_sandbox_auth_command take env= and a command= override for the probe. Integration: _run_warm names its container, and the two tests that discarded the warm result now assert it. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0126UoDvKj2BN5SHLH81aqnW --- src/helix/sandbox.py | 240 ++++++++++++++-- .../test_credential_warm_docker.py | 20 +- tests/unit/test_credential_warm.py | 257 ++++++++++++++++-- 3 files changed, 467 insertions(+), 50 deletions(-) diff --git a/src/helix/sandbox.py b/src/helix/sandbox.py index 020a61f9..eba0ecd1 100644 --- a/src/helix/sandbox.py +++ b/src/helix/sandbox.py @@ -2,12 +2,13 @@ from __future__ import annotations -from collections.abc import Iterator, Sequence +from collections.abc import Iterator, Mapping, Sequence from contextlib import contextmanager from dataclasses import dataclass import json import logging import os +from datetime import datetime, timedelta, timezone import shlex import shutil import subprocess @@ -1146,13 +1147,25 @@ def sandbox_auth_docker_args( extra_hosts: dict[str, str] | None = None, interactive: bool = False, container_name: str | None = None, + env: Mapping[str, str] | None = None, + command: Sequence[str] | None = None, ) -> list[str]: - try: - command = BACKEND_AUTH_COMMANDS[agent_backend][action] - except KeyError as exc: - raise ValueError( - f"No sandbox auth {action!r} command for backend: {agent_backend}" - ) from exc + """Build the ``docker run`` argv for one auth-related command. + + *env* is forwarded with ``-e`` (``HOME`` and ``PATH`` stay pinned); it is + how the operator's ``passthrough_env`` / ``[env]`` proxy and CA settings + reach a container that has to talk to the token endpoint. *command* + replaces the registered *action* argv when given -- used for read-only + probes against the login volume that are not auth commands in their own + right. + """ + if command is None: + try: + command = BACKEND_AUTH_COMMANDS[agent_backend][action] + except KeyError as exc: + raise ValueError( + f"No sandbox auth {action!r} command for backend: {agent_backend}" + ) from exc args = [ "docker", @@ -1176,6 +1189,10 @@ def sandbox_auth_docker_args( "-e", "PATH=/usr/local/sbin:/usr/local/bin:/usr/sbin:/usr/bin:/sbin:/bin", ] + for key, value in (env or {}).items(): + if key in {"HOME", "PATH"}: + continue + args.extend(["-e", f"{key}={value}"]) if container_name: args.extend(["--name", container_name]) if interactive: @@ -1196,6 +1213,8 @@ def run_sandbox_auth_command( interactive: bool = False, timeout: float | None = None, container_name: str | None = None, + env: Mapping[str, str] | None = None, + command: Sequence[str] | None = None, ) -> subprocess.CompletedProcess[str]: """Run one auth-related command against the shared login volume. @@ -1218,6 +1237,8 @@ def run_sandbox_auth_command( extra_hosts=extra_hosts, interactive=interactive, container_name=container_name, + env=env, + command=command, ) if interactive: return subprocess.run(args, text=True) @@ -1235,9 +1256,17 @@ def run_sandbox_auth_command( class CredentialWarmResult: """Outcome of one per-generation credential warm. - ``warmed`` is True only when the warm container ran and exited cleanly. + ``warmed`` is True only when the warm container exited cleanly **and** the + stored credential was verified fresh afterwards -- for codex, that its + ``last_refresh`` either advanced during the warm or already sits inside + the CLI's refresh interval, so no candidate will attempt a refresh. The + exit code alone cannot say that: ``codex debug models`` swallows a + rejected refresh and exits 0. + ``skip_reason`` is set when the backend is deliberately not warmed; - ``detail`` carries the diagnosis when a warm was attempted and failed. + ``detail`` carries the diagnosis when a warm was attempted, whether it + succeeded or not; ``stale`` marks a clean exit whose credential still + failed verification. """ backend: str @@ -1246,6 +1275,7 @@ class CredentialWarmResult: returncode: int | None = None detail: str = "" timed_out: bool = False + stale: bool = False @property def skipped(self) -> bool: @@ -1269,6 +1299,31 @@ def failed(self) -> bool: #: can only tighten it. CREDENTIAL_WARM_TIMEOUT_SECONDS = 300 +#: Bound on one read of ``last_refresh`` from the login volume: a ``sed`` over +#: one file in a container with no network. +_CREDENTIAL_PROBE_TIMEOUT_SECONDS = 60 + +#: How old a codex credential may be before the CLI refreshes it proactively. +#: ``TOKEN_REFRESH_INTERVAL: i64 = 8`` (days) in codex-rs +#: ``login/src/auth/manager.rs``: ``should_refresh_proactively`` returns true +#: when ``last_refresh < now - 8 days``. Newer builds also refresh when the +#: access token's JWT ``exp`` is within 5 minutes; that clock is not read +#: here, so a credential inside the 8-day interval whose access token is about +#: to expire is reported fresh although a refresh is imminent -- the +#: interval, not the JWT, is what the shipped measurement used. +CODEX_TOKEN_REFRESH_INTERVAL = timedelta(days=8) + +#: Extracts the ``last_refresh`` value -- and nothing else -- from codex's +#: ``auth.json``. The tokens beside it never leave the container. Prints +#: nothing when the file is absent. +_SED_LAST_REFRESH = r's/.*"last_refresh"[[:space:]]*:[[:space:]]*"\([^"]*\)".*/\1/p' +_CODEX_LAST_REFRESH_PROBE: tuple[str, ...] = ( + "sh", + "-c", + 'f="$HOME/.codex/auth.json"; [ -f "$f" ] || exit 0; ' + "sed -n '" + _SED_LAST_REFRESH + "' \"$f\"", +) + def credential_warm_timeout(sandbox: SandboxConfig) -> float: """Return the timeout for one warm: the fixed cap, tightened by the sandbox's.""" @@ -1277,8 +1332,118 @@ def credential_warm_timeout(sandbox: SandboxConfig) -> float: return float(CREDENTIAL_WARM_TIMEOUT_SECONDS) +def _parse_last_refresh(text: str) -> datetime | None: + value = text.strip().splitlines()[-1].strip() if text.strip() else "" + if not value: + return None + try: + parsed = datetime.fromisoformat(value.replace("Z", "+00:00")) + except ValueError: + return None + if parsed.tzinfo is None: + parsed = parsed.replace(tzinfo=timezone.utc) + return parsed + + +def _read_codex_last_refresh( + *, image: str, sandbox: SandboxConfig, timeout: float +) -> datetime | None: + """Read ``last_refresh`` from the codex login volume, or ``None``. + + Runs under ``--network none`` through the same single-writer auth + container as the warm; reads one field and never the tokens. ``None`` + means the value could not be read (no file, no field, container failure) + and is reported as "unverified" by the caller, never as fresh. + """ + container_name = f"helix-probe-codex-{uuid.uuid4().hex[:12]}" + try: + result = run_sandbox_auth_command( + "codex", + action="status", + image=image, + network="none", + add_host_gateway=sandbox.add_host_gateway, + extra_hosts=sandbox.extra_hosts, + timeout=min(timeout, _CREDENTIAL_PROBE_TIMEOUT_SECONDS), + container_name=container_name, + command=_CODEX_LAST_REFRESH_PROBE, + ) + except (OSError, ValueError, subprocess.SubprocessError) as exc: + logger.debug("could not read codex last_refresh: %s", exc) + return None + if result.returncode != 0: + logger.debug( + "codex last_refresh probe exited %d: %s", + result.returncode, + (result.stderr or "")[-_WARM_DETAIL_CHARS:], + ) + return None + return _parse_last_refresh(result.stdout or "") + + +def _describe_age(age: timedelta) -> str: + total = int(age.total_seconds()) + if total < 0: + return "in the future" + days, rem = divmod(total, 86400) + hours, rem = divmod(rem, 3600) + minutes = rem // 60 + if days: + return f"{days}d {hours}h" + if hours: + return f"{hours}h {minutes}m" + return f"{minutes}m" + + +def verify_codex_credential_fresh( + before: datetime | None, after: datetime | None, *, now: datetime +) -> tuple[bool, str]: + """Decide whether a codex warm left the credential verifiably fresh. + + Returns ``(fresh, detail)``. Fresh means no candidate in the coming + generation will attempt a refresh: either the warm performed one + (``last_refresh`` advanced) or none was due (``last_refresh`` is inside + :data:`CODEX_TOKEN_REFRESH_INTERVAL`). A ``last_refresh`` that is past + the interval and did not move means the exchange was rejected -- ``codex + debug models`` exits 0 either way -- or that a ``models_cache.json`` + younger than its 5-minute TTL short-circuited the refresh path; both + leave every candidate to attempt the refresh itself. + """ + if after is None: + return ( + False, + "the warm exited 0 but last_refresh could not be read back from " + "the login volume, so the credential is unverified", + ) + age = now - after + if before is not None and after > before: + return ( + True, + "refresh performed: last_refresh advanced to " + f"{after.isoformat(timespec='seconds')}", + ) + if age < CODEX_TOKEN_REFRESH_INTERVAL: + return ( + True, + f"credential fresh: last_refresh is {_describe_age(age)} old, inside " + f"codex's {CODEX_TOKEN_REFRESH_INTERVAL.days}-day refresh interval, " + "so no candidate will refresh it", + ) + return ( + False, + f"last_refresh is {_describe_age(age)} old, past codex's " + f"{CODEX_TOKEN_REFRESH_INTERVAL.days}-day refresh interval, and did not " + "advance during the warm: the token exchange was most likely rejected " + "(codex debug models exits 0 either way), or a models_cache.json " + "younger than 5 minutes short-circuited the refresh", + ) + + def warm_backend_credential( - agent_backend: str, *, sandbox: SandboxConfig + agent_backend: str, + *, + sandbox: SandboxConfig, + env: Mapping[str, str] | None = None, ) -> CredentialWarmResult: """Refresh *agent_backend*'s shared credential once, under a single writer. @@ -1289,10 +1454,15 @@ def warm_backend_credential( written back before candidates start, instead of N candidates racing to spend the same single-use refresh token. - The call is a no-op whenever the credential is already fresh -- the warm - command is chosen so the CLI does no work in that case (see - ``helix.backends``). Backends that need no warm return a skipped result - rather than starting a container. + *env* is the operator's ``passthrough_env`` / ``[env]`` selection, so the + warm reaches the token endpoint through the same proxy and CA settings + the candidates get. + + ``warmed`` means the credential was **verified** fresh after the warm, + not merely that the command exited 0 (see :class:`CredentialWarmResult`). + For codex, ``last_refresh`` is read from ``auth.json`` before and after + -- one field, no network -- and judged by + :func:`verify_codex_credential_fresh`. Never raises: a warm that cannot run is reported, not fatal. Candidates may still succeed on the credential that is already there, so the run @@ -1310,6 +1480,16 @@ def warm_backend_credential( container_name = f"helix-warm-{agent_backend}-{uuid.uuid4().hex[:12]}" try: image = resolve_sandbox_image(sandbox, agent_backend) + except ValueError as exc: + return CredentialWarmResult( + backend=agent_backend, warmed=False, detail=f"ValueError: {exc}" + ) + before = ( + _read_codex_last_refresh(image=image, sandbox=sandbox, timeout=timeout) + if agent_backend == "codex" + else None + ) + try: result = run_sandbox_auth_command( agent_backend, action="warm", @@ -1319,6 +1499,7 @@ def warm_backend_credential( extra_hosts=sandbox.extra_hosts, timeout=timeout, container_name=container_name, + env=env, ) except subprocess.TimeoutExpired: # Non-fatal, like every other warm failure: the candidates fall back @@ -1340,16 +1521,35 @@ def warm_backend_credential( detail=f"{type(exc).__name__}: {exc}", ) - if result.returncode == 0: + if result.returncode != 0: + stderr = (result.stderr or "").strip() + return CredentialWarmResult( + backend=agent_backend, + warmed=False, + returncode=result.returncode, + detail=stderr[-_WARM_DETAIL_CHARS:], + ) + + if agent_backend != "codex": + # Only codex has a warm command today; a future backend needs its own + # verifier before its exit code may be trusted. return CredentialWarmResult( - backend=agent_backend, warmed=True, returncode=0 + backend=agent_backend, + warmed=False, + returncode=0, + stale=True, + detail="the warm exited 0 but no freshness verifier exists for this backend", ) - stderr = (result.stderr or "").strip() + after = _read_codex_last_refresh(image=image, sandbox=sandbox, timeout=timeout) + fresh, detail = verify_codex_credential_fresh( + before, after, now=datetime.now(timezone.utc) + ) return CredentialWarmResult( backend=agent_backend, - warmed=False, - returncode=result.returncode, - detail=stderr[-_WARM_DETAIL_CHARS:], + warmed=fresh, + returncode=0, + stale=not fresh, + detail=detail, ) diff --git a/tests/integration/test_credential_warm_docker.py b/tests/integration/test_credential_warm_docker.py index 3b74cd84..d0eabfd2 100644 --- a/tests/integration/test_credential_warm_docker.py +++ b/tests/integration/test_credential_warm_docker.py @@ -6,11 +6,13 @@ (a) it costs nothing -- no model call, no quota, on an operator's paid account that would otherwise be charged once per generation; and -(b) it does nothing at all when the credential is already fresh. +(b) it does not touch the credential when it is already fresh. Both are asserted here by running the registered warm command with ``--network none``. A command that completes with no network cannot have -reached a model; a warm that leaves the volume byte-identical did no work. +reached a model; a warm that leaves ``auth.json`` byte-identical spent no +refresh token. (The CLI may still write its own catalog cache, +``~/.codex/models_cache.json``, beside it; that file is not the credential.) Credentials are synthetic and live in throwaway volumes (see ``conftest``); no test logs in, and none can reach a real ``helix-auth-*`` volume. @@ -19,6 +21,7 @@ from __future__ import annotations import subprocess +import uuid import pytest @@ -67,6 +70,10 @@ def _run_warm(*, image: str, volume: str, backend: str, timeout: int = 180): "docker", "run", "--rm", + # Named so a hang can be stopped by name; the volume fixture removes + # its volume in teardown, which fails while a container holds it. + "--name", + f"helix-integration-warm-{uuid.uuid4().hex[:12]}", "--network", "none", "--security-opt", @@ -161,8 +168,9 @@ def test_codex_warm_is_a_no_op_on_a_fresh_credential( before_files = _files(volume_listing(volume, image)) before_digest = _auth_digest(volume, image) - _run_warm(image=image, volume=volume, backend="codex") + result = _run_warm(image=image, volume=volume, backend="codex") + assert result.returncode == 0, result.stderr assert _auth_digest(volume, image) == before_digest assert _files(volume_listing(volume, image)) == before_files @@ -181,12 +189,14 @@ def test_repeated_codex_warms_leave_no_residue( image = require_image(CODEX_IMAGE) volume = throwaway_volume(image, SYNTHETIC_FRESH_CODEX_AUTH) - _run_warm(image=image, volume=volume, backend="codex") + first = _run_warm(image=image, volume=volume, backend="codex") + assert first.returncode == 0, first.stderr after_first = volume_listing(volume, image) digest_first = _auth_digest(volume, image) for _ in range(2): - _run_warm(image=image, volume=volume, backend="codex") + again = _run_warm(image=image, volume=volume, backend="codex") + assert again.returncode == 0, again.stderr assert volume_listing(volume, image) == after_first assert _auth_digest(volume, image) == digest_first diff --git a/tests/unit/test_credential_warm.py b/tests/unit/test_credential_warm.py index ddad6c2f..04828e90 100644 --- a/tests/unit/test_credential_warm.py +++ b/tests/unit/test_credential_warm.py @@ -15,6 +15,7 @@ from __future__ import annotations import subprocess +from datetime import datetime, timedelta, timezone from typing import Any import pytest @@ -25,13 +26,21 @@ CREDENTIAL_WARM_SKIP_REASONS, backend_credential_warm_skip_reason, ) -from helix.config import AgentConfig, EvaluatorConfig, HelixConfig, SandboxConfig +from helix.config import ( + AgentConfig, + EvaluatorConfig, + EvolutionConfig, + HelixConfig, + SandboxConfig, +) from helix.evolution import _warm_generation_credential from helix.sandbox import ( + CODEX_TOKEN_REFRESH_INTERVAL, CREDENTIAL_WARM_TIMEOUT_SECONDS, CredentialWarmResult, credential_warm_timeout, sandbox_auth_docker_args, + verify_codex_credential_fresh, warm_backend_credential, ) @@ -40,12 +49,75 @@ SKIPPED_BACKENDS = ("agy", "claude", "cursor", "opencode") -def _completed(returncode: int, stderr: str = "") -> subprocess.CompletedProcess[str]: +def _completed( + returncode: int, stderr: str = "", stdout: str = "" +) -> subprocess.CompletedProcess[str]: return subprocess.CompletedProcess( - args=["docker"], returncode=returncode, stdout="", stderr=stderr + args=["docker"], returncode=returncode, stdout=stdout, stderr=stderr ) +NOW = datetime(2026, 9, 10, 12, 0, 0, tzinfo=timezone.utc) + + +def _stamp(dt: datetime) -> str: + return dt.strftime("%Y-%m-%dT%H:%M:%S.%fZ") + "\n" + + +class _FakeAuthCommands: + """Stand-in for ``run_sandbox_auth_command`` that answers the + ``last_refresh`` probes from a script and records the warm call. + + ``timeline`` is the sequence of values the probe returns (before, after); + ``warm`` is the warm command's result. + """ + + def __init__( + self, + timeline: list[str | None], + warm: subprocess.CompletedProcess[str] | BaseException | None = None, + ) -> None: + self.timeline = list(timeline) + self.warm = warm if warm is not None else _completed(0) + self.calls: list[dict[str, Any]] = [] + + def __call__(self, backend: str, **kwargs: Any) -> subprocess.CompletedProcess[str]: + record = dict(kwargs) + record["backend"] = backend + self.calls.append(record) + if kwargs.get("command") is not None: + value = self.timeline.pop(0) + return _completed(0, stdout="" if value is None else value) + if isinstance(self.warm, BaseException): + raise self.warm + return self.warm + + @property + def warm_calls(self) -> list[dict[str, Any]]: + return [c for c in self.calls if c.get("command") is None] + + @property + def probe_calls(self) -> list[dict[str, Any]]: + return [c for c in self.calls if c.get("command") is not None] + + +def _fresh_timeline() -> list[str | None]: + """A credential refreshed an hour ago: nothing to do, verifiably fresh.""" + stamp = _stamp(NOW - timedelta(hours=1)) + return [stamp, stamp] + + +def _install(monkeypatch: pytest.MonkeyPatch, fake: _FakeAuthCommands) -> None: + monkeypatch.setattr("helix.sandbox.run_sandbox_auth_command", fake) + monkeypatch.setattr("helix.sandbox.datetime", _FrozenDatetime) + + +class _FrozenDatetime(datetime): + @classmethod + def now(cls, tz: Any = None) -> "datetime": # type: ignore[override] + return NOW if tz is not None else NOW.replace(tzinfo=None) + + # --------------------------------------------------------------------------- # Registry: every backend is either warmed or explained # --------------------------------------------------------------------------- @@ -122,14 +194,8 @@ def test_warm_action_is_rejected_for_a_backend_without_one(self) -> None: def test_warm_forwards_sandbox_network_settings( self, monkeypatch: pytest.MonkeyPatch ) -> None: - seen: dict[str, Any] = {} - - def _fake(backend: str, **kwargs: Any) -> subprocess.CompletedProcess[str]: - seen.update(kwargs) - seen["backend"] = backend - return _completed(0) - - monkeypatch.setattr("helix.sandbox.run_sandbox_auth_command", _fake) + fake = _FakeAuthCommands(_fresh_timeline()) + _install(monkeypatch, fake) sandbox = SandboxConfig( enabled=True, network="none", @@ -140,6 +206,7 @@ def _fake(backend: str, **kwargs: Any) -> subprocess.CompletedProcess[str]: result = warm_backend_credential("codex", sandbox=sandbox) assert result.warmed is True + [seen] = fake.warm_calls assert seen["backend"] == "codex" assert seen["action"] == "warm" assert seen["network"] == "none" @@ -149,6 +216,56 @@ def _fake(backend: str, **kwargs: Any) -> subprocess.CompletedProcess[str]: # with a different CLI build than the candidates use. assert seen["image"] == "custom:tag" + def test_operator_env_reaches_the_warm_container( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + """Proxy / CA settings the candidates get must reach the warm too, or + it cannot reach the token endpoint the candidates can.""" + fake = _FakeAuthCommands(_fresh_timeline()) + _install(monkeypatch, fake) + env = {"HTTPS_PROXY": "http://proxy:3128", "SSL_CERT_FILE": "/ca.pem"} + warm_backend_credential( + "codex", sandbox=SandboxConfig(enabled=True), env=env + ) + [seen] = fake.warm_calls + assert seen["env"] == env + args = sandbox_auth_docker_args( + "codex", image="img:latest", action="warm", env=env + ) + joined = " ".join(args) + assert "-e HTTPS_PROXY=http://proxy:3128" in joined + assert "-e SSL_CERT_FILE=/ca.pem" in joined + # HOME and PATH stay pinned to the container's own values. + assert "-e HOME=/home/node" in joined + assert sandbox_auth_docker_args( + "codex", image="img", action="warm", env={"HOME": "/x", "PATH": "/y"} + ).count("-e") == 2 + + def test_no_auth_command_uses_a_login_shell(self) -> None: + """``sh -l`` sources ``$HOME/.profile`` from the shared login volume + that every candidate can write; PATH is pinned with ``-e`` instead.""" + for backend, actions in BACKEND_AUTH_COMMANDS.items(): + for action, argv in actions.items(): + if argv[0] != "sh": + continue + assert argv[1] == "-c", (backend, action, argv) + assert not any( + flag.startswith("-") and "l" in flag for flag in argv[1:-1] + ), (backend, action, argv) + + def test_last_refresh_probe_runs_without_network_and_reads_one_field( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + fake = _FakeAuthCommands(_fresh_timeline()) + _install(monkeypatch, fake) + warm_backend_credential("codex", sandbox=SandboxConfig(enabled=True)) + before, after = fake.probe_calls + for probe in (before, after): + assert probe["network"] == "none" + script = probe["command"][-1] + assert "last_refresh" in script + assert "access_token" not in script and "cat " not in script + # --------------------------------------------------------------------------- # Failure of the warm is reported, never fatal @@ -173,10 +290,8 @@ def _boom(*_a: Any, **_k: Any) -> None: def test_non_zero_exit_is_reported_not_raised( self, monkeypatch: pytest.MonkeyPatch ) -> None: - monkeypatch.setattr( - "helix.sandbox.run_sandbox_auth_command", - lambda *a, **k: _completed(3, "warm blew up"), - ) + fake = _FakeAuthCommands(_fresh_timeline(), warm=_completed(3, "warm blew up")) + _install(monkeypatch, fake) result = warm_backend_credential( "codex", sandbox=SandboxConfig(enabled=True) ) @@ -187,10 +302,8 @@ def test_non_zero_exit_is_reported_not_raised( def test_docker_exception_is_reported_not_raised( self, monkeypatch: pytest.MonkeyPatch ) -> None: - def _raise(*_a: Any, **_k: Any) -> None: - raise OSError("no docker here") - - monkeypatch.setattr("helix.sandbox.run_sandbox_auth_command", _raise) + fake = _FakeAuthCommands([None, None], warm=OSError("no docker here")) + _install(monkeypatch, fake) result = warm_backend_credential( "codex", sandbox=SandboxConfig(enabled=True) ) @@ -198,14 +311,104 @@ def _raise(*_a: Any, **_k: Any) -> None: assert "no docker here" in result.detail def test_detail_is_capped(self, monkeypatch: pytest.MonkeyPatch) -> None: + fake = _FakeAuthCommands(_fresh_timeline(), warm=_completed(1, "x" * 5000)) + _install(monkeypatch, fake) + result = warm_backend_credential( + "codex", sandbox=SandboxConfig(enabled=True) + ) + assert 0 < len(result.detail) <= 400 + + +# --------------------------------------------------------------------------- +# ``warmed`` means verified fresh, not "exited 0" +# --------------------------------------------------------------------------- + + +class TestWarmVerifiesFreshness: + """``codex debug models`` swallows a rejected refresh and exits 0. + + The exit code therefore proves nothing. ``last_refresh`` is read back + from ``auth.json`` (one field, no network) and the warm is reported as + such only when the credential is verifiably inside codex's refresh + interval afterwards -- so no candidate will attempt a refresh. + """ + + def test_advanced_last_refresh_is_warmed( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + stale = _stamp(NOW - CODEX_TOKEN_REFRESH_INTERVAL - timedelta(days=1)) + fake = _FakeAuthCommands([stale, _stamp(NOW - timedelta(seconds=2))]) + _install(monkeypatch, fake) + result = warm_backend_credential( + "codex", sandbox=SandboxConfig(enabled=True) + ) + assert result.warmed is True + assert result.stale is False + assert "refresh performed" in result.detail + + def test_stale_and_unchanged_is_not_warmed( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + """A rejected refresh: exit 0, ``last_refresh`` still past the interval.""" + stale = _stamp(NOW - CODEX_TOKEN_REFRESH_INTERVAL - timedelta(days=1)) + fake = _FakeAuthCommands([stale, stale]) + _install(monkeypatch, fake) + result = warm_backend_credential( + "codex", sandbox=SandboxConfig(enabled=True) + ) + assert result.warmed is False + assert result.stale is True + assert result.failed is True + assert result.returncode == 0 + assert "rejected" in result.detail + assert "models_cache.json" in result.detail + + # The loop turns it into a WARNING that names the cause and the race. monkeypatch.setattr( - "helix.sandbox.run_sandbox_auth_command", - lambda *a, **k: _completed(1, "x" * 5000), + "helix.evolution.warm_backend_credential", lambda backend, **_k: result + ) + _warm_generation_credential( + _config("codex", sandboxed=True), gen=2, announce_skip=False ) + out = " ".join(capsys.readouterr().out.lower().split()) + assert "not verifiably fresh" in out + assert "rejected" in out + assert "same single-use refresh token" in out + + def test_unchanged_but_inside_the_interval_is_fresh( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + """No refresh was due (or a fresh catalog cache short-circuited one): + either way no candidate will refresh, so there is nothing to race.""" + fake = _FakeAuthCommands(_fresh_timeline()) + _install(monkeypatch, fake) result = warm_backend_credential( "codex", sandbox=SandboxConfig(enabled=True) ) - assert 0 < len(result.detail) <= 400 + assert result.warmed is True + assert "inside codex's 8-day refresh interval" in result.detail + + def test_unreadable_last_refresh_is_unverified_not_warmed( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + fake = _FakeAuthCommands([None, None]) + _install(monkeypatch, fake) + result = warm_backend_credential( + "codex", sandbox=SandboxConfig(enabled=True) + ) + assert result.warmed is False + assert result.stale is True + assert "unverified" in result.detail + + def test_verdict_function(self) -> None: + old = NOW - CODEX_TOKEN_REFRESH_INTERVAL - timedelta(hours=1) + recent = NOW - timedelta(minutes=5) + assert verify_codex_credential_fresh(old, recent, now=NOW)[0] is True + assert verify_codex_credential_fresh(old, old, now=NOW)[0] is False + assert verify_codex_credential_fresh(recent, recent, now=NOW)[0] is True + assert verify_codex_credential_fresh(None, recent, now=NOW)[0] is True + assert verify_codex_credential_fresh(None, None, now=NOW)[0] is False + assert verify_codex_credential_fresh(old, None, now=NOW)[0] is False # --------------------------------------------------------------------------- @@ -241,11 +444,14 @@ def test_warm_runs_with_a_timeout_and_a_named_container( seen: dict[str, Any] = {} def _fake_run(args: list[str], **kwargs: Any) -> subprocess.CompletedProcess[str]: + if "last_refresh" in args[-1]: + return _completed(0, stdout=_stamp(NOW - timedelta(hours=1))) seen["args"] = args seen.update(kwargs) return _completed(0) monkeypatch.setattr("helix.sandbox.subprocess.run", _fake_run) + monkeypatch.setattr("helix.sandbox.datetime", _FrozenDatetime) result = warm_backend_credential( "codex", sandbox=SandboxConfig(enabled=True, timeout_seconds=45) ) @@ -281,9 +487,10 @@ def _fake_docker(args: list[str], **_k: Any) -> subprocess.CompletedProcess[str] assert "7s" in result.detail # Killing the docker client does not stop the container; the named # container must be force-removed so it stops touching the volume. - assert len(removed) == 1 - assert removed[0][:3] == ["docker", "rm", "-f"] - assert removed[0][3].startswith("helix-warm-codex-") + # (The probe before the warm hung too under this fake and was removed + # the same way.) + assert all(r[:3] == ["docker", "rm", "-f"] for r in removed) + assert any(r[3].startswith("helix-warm-codex-") for r in removed) def test_timed_out_warm_is_a_warning_in_the_loop( self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] From a6c67948c7812ee4b138e72db531d50159290386 Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Thu, 10 Sep 2026 21:10:13 -0700 Subject: [PATCH 15/16] evolution: gate the warm on concurrency, escape container output, one credential diagnosis per site Item 12: the warm is skipped, with the reason logged once, when at most one candidate can write the shared login in a generation (min(num_parallel_proposals * mutations_per_parent, max_workers) <= 1); a single writer cannot race itself. Every "starts from an already-fresh credential" claim is softened to fresh at the start of the generation -- a token that expires during it can still race. Cadence stays per-generation. Item 5: the warm's stderr tail, the summary's last report and the worker-exception text are passed through rich.markup.escape, so a tail containing [/] or [/bold] no longer raises MarkupError on the main thread. Item 11: a merge that failed on a credential error no longer falls into the generic "returned no output" message; the merge-site wording honours transient like the worker; CredentialFailureLog records the kind so the summary says "1 merge(s)"; the operator-facing sentence and remedy live in _credential_failure_verdict / _credential_remedy instead of nine literals. Recovered refresh races (from mutate/merge's on_refresh_race_recovered) are listed in the end-of-run summary. Item 10: the CLI handler only promises 'helix resume' when .helix/state.json exists (state_file_exists) and otherwise says no state was saved and points at 'helix evolve'; the seedless seed path charges exc.usage (source=seed_generation_failed) before re-raising, mirroring merger.py. Tests: single-writer skip and two-writer warm, operator env reaching the warm, markup in the stderr tail, exactly one merge-site diagnosis with transient wording and a merge(s) label, the recovered race in the summary, CLI hint with and without state.json, seed usage charged. The loop tests now warm codex, the backend the code actually warms. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0126UoDvKj2BN5SHLH81aqnW --- src/helix/backends.py | 12 +- src/helix/cli.py | 42 +++- src/helix/evolution.py | 285 +++++++++++++++++----- src/helix/state.py | 5 + tests/unit/test_cli_credential_failure.py | 31 ++- tests/unit/test_credential_warm.py | 113 ++++++++- tests/unit/test_credential_warm_loop.py | 116 ++++++++- tests/unit/test_evolution_seedless.py | 23 ++ 8 files changed, 541 insertions(+), 86 deletions(-) diff --git a/src/helix/backends.py b/src/helix/backends.py index c90d9afa..f41dc897 100644 --- a/src/helix/backends.py +++ b/src/helix/backends.py @@ -201,11 +201,15 @@ # independently that a refresh is due, and each posts the same single-use # refresh token. One wins; the rest are told the token was already consumed. # -# ``helix.sandbox.warm_backend_credential`` closes that window by running the +# ``helix.sandbox.warm_backend_credential`` narrows that window by running the # command below once, in one container, before a generation dispatches any -# candidate -- so whatever refresh is due happens under a single writer and -# every candidate then starts from an already-fresh credential with nothing -# left to race for. +# candidate -- so a refresh that is due at that moment happens under a single +# writer. The credential is then fresh at the *start* of the generation; a +# token that crosses its refresh threshold during the generation (parent +# evaluations run before each mutation, and queued slots start later still) +# can still be raced by the candidates in flight. The warm is skipped when +# at most one candidate can write the shared login at a time, since a single +# writer cannot race itself. # # A backend is warmed only when a command exists here that (a) actually takes # the CLI's refresh path and (b) costs nothing. Both halves are load-bearing: diff --git a/src/helix/cli.py b/src/helix/cli.py index 49addb15..456f2420 100644 --- a/src/helix/cli.py +++ b/src/helix/cli.py @@ -34,7 +34,7 @@ ) from helix.lineage import load_lineage from helix.population import EvalResult, FrontierType, ParetoFrontier, Candidate -from helix.state import load_state, save_state +from helix.state import load_state, save_state, state_file_exists from helix.worktree import remove_worktree logger = logging.getLogger(__name__) @@ -114,6 +114,28 @@ def _helix_dir(project_root: Path) -> Path: return project_root / _HELIX_DIR +def _print_credential_failure_hint( + project_root: Path, backend: str, exc: CredentialRefreshError +) -> None: + """Say what to do after a credential failure -- truthfully about state. + + Seedless seed generation is the one path that reaches the CLI handler, + and it fails before the first ``save_state``; promising ``helix resume`` + then sends the operator to a command that starts a fresh run. + """ + from helix.evolution import _credential_remedy + + remedy = _credential_remedy(backend, transient=exc.transient) + if state_file_exists(project_root): + print_error(f"Evolution state has been saved. {remedy}") + return + print_error( + "No evolution state was saved: the failure happened before the first " + "generation completed, so there is nothing to resume. " + + remedy.replace("[cyan]helix resume[/cyan]", "[cyan]helix evolve[/cyan]") + ) + + def _print_cleanup_hint() -> None: print_info( "HELIX worktrees and saved state remain on disk after the run. " @@ -702,15 +724,13 @@ def evolve( raise SystemExit(2) except CredentialRefreshError as exc: # Every in-loop path handles this itself (the slot is skipped and the - # run continues), so reaching here means a path that does not. Show - # the panel with its suggestion instead of a raw traceback. + # run continues), so reaching here means a path that does not -- in + # practice seedless seed generation, which runs before any state has + # been saved. Show the panel with its suggestion instead of a raw + # traceback, and only promise a resume when there is a state file. logger.error("Credential failure escaped the evolution loop: %s", exc) print_helix_error(exc) - print_error( - "Evolution state has been saved. Re-authenticate with " - f"[cyan]helix sandbox login {config.agent.backend}[/cyan] if the " - "login is stale, then run [cyan]helix resume[/cyan]." - ) + _print_credential_failure_hint(project_root, config.agent.backend, exc) raise SystemExit(2) except KeyboardInterrupt: _handle_keyboard_interrupt(project_root) @@ -1257,11 +1277,7 @@ def resume(config_path: str, project_dir: Path | None) -> None: except CredentialRefreshError as exc: logger.error("Credential failure escaped the resumed loop: %s", exc) print_helix_error(exc) - print_error( - "Evolution state has been saved. Re-authenticate with " - f"[cyan]helix sandbox login {config.agent.backend}[/cyan] if the " - "login is stale, then run [cyan]helix resume[/cyan] again." - ) + _print_credential_failure_hint(project_root, config.agent.backend, exc) raise SystemExit(2) except KeyboardInterrupt: _handle_keyboard_interrupt(project_root) diff --git a/src/helix/evolution.py b/src/helix/evolution.py index fc7fdd3a..086f0e91 100644 --- a/src/helix/evolution.py +++ b/src/helix/evolution.py @@ -18,6 +18,8 @@ from pathlib import Path from typing import Any +from rich.markup import escape + from helix.batch_sampler import ( BatchSampler, @@ -1461,16 +1463,30 @@ class CredentialFailureLog: lock. The log exists so a run that dies from an unusable login says so once, plainly, in the permanent end-of-run summary -- not only in a per-slot error that has already scrolled past by the time the run ends. + + ``recovered`` holds the invocations that lost a refresh race and + succeeded on their one retry: not failures, but the only evidence an + operator gets that the warm is not protecting the run. """ - entries: list[tuple[str, str, bool]] = field(default_factory=list) + entries: list[tuple[str, str, bool, str]] = field(default_factory=list) + recovered: list[tuple[str, str]] = field(default_factory=list) _lock: threading.Lock = field(default_factory=threading.Lock, repr=False) def record( - self, candidate_id: str, message: str, *, transient: bool = False + self, + candidate_id: str, + message: str, + *, + transient: bool = False, + kind: str = "mutation", ) -> None: with self._lock: - self.entries.append((candidate_id, message, transient)) + self.entries.append((candidate_id, message, transient, kind)) + + def record_recovered(self, candidate_id: str, message: str) -> None: + with self._lock: + self.recovered.append((candidate_id, message)) def __len__(self) -> int: with self._lock: @@ -1478,7 +1494,11 @@ def __len__(self) -> int: def candidate_ids(self) -> list[str]: with self._lock: - return [candidate_id for candidate_id, _, _ in self.entries] + return [candidate_id for candidate_id, _, _, _ in self.entries] + + def recovered_ids(self) -> list[str]: + with self._lock: + return [candidate_id for candidate_id, _ in self.recovered] def last_message(self) -> str: with self._lock: @@ -1487,7 +1507,81 @@ def last_message(self) -> str: def all_transient(self) -> bool: """True when every failure was a lost refresh race, not a dead login.""" with self._lock: - return bool(self.entries) and all(t for _, _, t in self.entries) + return bool(self.entries) and all(t for _, _, t, _ in self.entries) + + def describe_failed(self) -> str: + """``"2 mutation(s) and 1 merge(s)"`` -- what failed, by kind.""" + with self._lock: + kinds = [kind for _, _, _, kind in self.entries] + parts = [ + f"{kinds.count(kind)} {kind}(s)" + for kind in ("mutation", "merge") + if kind in kinds + ] + return " and ".join(parts) if parts else "0 invocation(s)" + + +def _credential_failure_verdict( + backend: str, *, transient: bool, subject: str +) -> str: + """The one operator-facing sentence for a credential failure. + + Used by the proposal worker and the merge gate so the cause is worded + once (the CLI and the summary share :func:`_credential_remedy`); + *subject* names what the failure is **not** about ("the candidate's + code", "the merged code"). + """ + display = backend_display_name(backend) + if transient: + cause = ( + f"the shared {display} credential was refreshed by another " + "candidate first and the retry also failed" + ) + else: + cause = f"the shared {display} credential could not be used or refreshed" + return ( + f"failed because {cause} — this is a login failure, " + f"not a failure of {subject}." + ) + + +def _credential_remedy(backend: str, *, transient: bool) -> str: + """What the operator should do about a credential failure.""" + if transient: + return ( + "The stored login is most likely usable: run " + "[cyan]helix resume[/cyan] first, and only re-authenticate if " + "this keeps recurring." + ) + return ( + f"Re-authenticate with [cyan]helix sandbox login {backend}[/cyan], " + "then [cyan]helix resume[/cyan]." + ) + + +def _generation_concurrent_writers(config: HelixConfig) -> int: + """How many candidates can write the shared login at once in a generation. + + ``num_parallel_proposals * mutations_per_parent`` slots, run under a pool + of at most ``max_workers``; a single slot takes the in-thread path with no + pool at all. The merge branch is one invocation. + """ + slots = ( + config.evolution.num_parallel_proposals + * config.evolution.mutations_per_parent + ) + return max(1, min(slots, config.evolution.max_workers)) + + +def _operator_env(config: HelixConfig) -> dict[str, str]: + """The operator's ``passthrough_env`` / ``[env]`` selection, as a mapping.""" + env = { + key: os.environ[key] + for key in config.passthrough_env + if key in os.environ + } + env.update(config.env) + return env def _warm_generation_credential( @@ -1496,32 +1590,58 @@ def _warm_generation_credential( """Refresh the agent backend's shared credential once for generation *gen*. Called once per generation, before any candidate is dispatched, so that a - refresh which has come due happens under one writer and every candidate in - the generation then starts from an already-fresh credential. + refresh which has come due happens under one writer and the credential is + fresh at the start of the generation. That is the extent of the + protection: a token that crosses its refresh threshold *during* the + generation -- each worker runs the parent evaluation before its mutation, + and slots beyond ``max_workers`` start later still -- can still be raced + by the candidates in flight. Per *generation* rather than once per run on purpose: a long run outlives any refresh interval, so a single warm at startup stops protecting the run the moment the credential next goes stale mid-flight. + Skipped when at most one candidate can write the shared login at a time + (``_generation_concurrent_writers``): a single writer cannot race itself, + and the warm would only add a container per generation. + Returns ``None`` when there is nothing to warm -- an unsandboxed run has no HELIX-managed login volume, because the backend runs directly against the operator's own CLI state and HELIX never mounts or arbitrates it. - Never fatal. A warm that could not run leaves exactly today's behaviour in - place (candidates refresh for themselves and may race), and candidates may - still succeed on the credential already stored -- so the run continues and - the operator is told, in those terms, what protection was lost. + Never fatal. A warm that could not run, or that left the credential + unverified, leaves exactly today's behaviour in place (candidates refresh + for themselves and may race), and candidates may still succeed on the + credential already stored -- so the run continues and the operator is + told, in those terms, what protection was lost. """ if not config.sandbox.enabled: return None backend = config.agent.backend display = backend_display_name(backend) - result = warm_backend_credential(backend, sandbox=config.sandbox) + writers = _generation_concurrent_writers(config) + if writers <= 1: + result = CredentialWarmResult( + backend=backend, + warmed=False, + skip_reason=( + "at most one candidate writes the shared login at a time " + f"(num_parallel_proposals × mutations_per_parent = " + f"{config.evolution.num_parallel_proposals * config.evolution.mutations_per_parent}, " + f"max_workers = {config.evolution.max_workers}); a single " + "writer cannot lose a refresh race to itself" + ), + ) + else: + result = warm_backend_credential( + backend, sandbox=config.sandbox, env=_operator_env(config) + ) if result.skipped: - # The reason is a property of the backend, not of this generation, so - # say it once per run instead of once per generation. + # The reason is a property of the backend or the run's concurrency, + # not of this generation, so say it once per run instead of once per + # generation. if announce_skip: logger.info( "No credential warm for %s: %s", display, result.skip_reason @@ -1530,13 +1650,18 @@ def _warm_generation_credential( if result.warmed: logger.debug( - "Credential warm for %s completed before generation %d.", display, gen + "Credential warm for %s completed before generation %d: %s", + display, + gen, + result.detail, ) return result - detail = f" Detail: {result.detail}" if result.detail else "" + detail = f" Detail: {escape(result.detail)}" if result.detail else "" if result.timed_out: cause = "timed out" + elif result.stale: + cause = "exited 0 but the credential is not verifiably fresh" elif result.returncode is not None: cause = f"exit {result.returncode}" else: @@ -1672,6 +1797,9 @@ def _run_proposal_worker( ) ), record_usage=_spent_usage.append, + on_refresh_race_recovered=lambda msg: ( + credential_failures.record_recovered(_new_id, msg) + ), ) except Exception as _mu_exc: # Re-raise PromptArtifactCollisionError (fatal for the whole run) @@ -1709,16 +1837,12 @@ def _run_proposal_worker( backend_display_name(config.agent.backend), _mu_exc, ) print_error( - f"Mutation [bold]{_new_id}[/bold] failed because the shared " - f"{backend_display_name(config.agent.backend)} credential " - + ( - "was refreshed by another candidate first and the " - "retry also failed" - if _mu_exc.transient - else "could not be used or refreshed" + f"Mutation [bold]{_new_id}[/bold] " + + _credential_failure_verdict( + config.agent.backend, + transient=_mu_exc.transient, + subject="the candidate's code", ) - + " — this is a login failure, not a failure of the " - "candidate's code." ) else: print_error( @@ -1823,7 +1947,8 @@ def _dispatch_proposals( f"Worker for proposal {_wid} " f"(parent: {_wparent.id}, gen {gen}) " f"raised an unexpected exception: " - f"{type(_wexc).__name__}: {_wexc} — proposal slot dropped." + f"{type(_wexc).__name__}: {escape(str(_wexc))} — " + "proposal slot dropped." ) worker_results[_widx] = MutationFailedProposal( presample_ctx=_wpctx, @@ -2222,7 +2347,18 @@ def _sync_frontier_state() -> None: candidate_id=seed.id, source="seed_generation", ) - except Exception: + except Exception as _seed_exc: + # The tokens the seed invocation spent before it failed are + # still spent; charge them before the worktree goes, exactly + # as ``merge()`` / ``mutate()`` hand a failed attempt's usage + # to their sink. + if isinstance(_seed_exc, HelixError) and _seed_exc.usage is not None: + budget_api.charge_llm_usage( + state, + _seed_exc.usage, + candidate_id=seed.id, + source="seed_generation_failed", + ) _safe_remove_worktree(seed, label="failed seed generation") raise print_success("Seed generation complete.") @@ -2368,9 +2504,10 @@ def _sync_frontier_state() -> None: # ---- Credential warm (once per generation) ------------------- # Sits above the merge/mutate split so it covers every path that - # dispatches a candidate this generation, and above every - # candidate so the shared login is already fresh by the time any - # of them could start refreshing it themselves. + # dispatches a candidate this generation. It makes the shared + # login fresh at the start of the generation; a token that + # crosses its refresh threshold while candidates are in flight + # can still be raced. Skipped when only one writer exists. _warm = _warm_generation_credential( config, gen=gen, announce_skip=not credential_warm_skip_announced ) @@ -2539,6 +2676,7 @@ def _has_val_support_overlap(i: str, j: str) -> bool: ) merge_usage: list[UsageStats] = [] + merge_credential_failed = False try: merged = merge( candidate_a=a, @@ -2556,17 +2694,22 @@ def _has_val_support_overlap(i: str, j: str) -> bool: ), ancestor=ancestor_candidate, record_usage=merge_usage.append, + on_refresh_race_recovered=lambda msg: ( + credential_failures.record_recovered(merge_id, msg) + ), ) except CredentialRefreshError as _merge_cred_exc: # Same treatment the proposal worker gives a # mutation: the merge worktree is already cleaned - # up by merge(); count and name the failure, then - # fall through to mutation so the run continues. + # up by merge(); count and name the failure once, + # then fall through to mutation so the run continues. merged = None + merge_credential_failed = True credential_failures.record( merge_id, str(_merge_cred_exc), transient=_merge_cred_exc.transient, + kind="merge", ) print_helix_error(_merge_cred_exc) logger.error( @@ -2577,11 +2720,13 @@ def _has_val_support_overlap(i: str, j: str) -> bool: _merge_cred_exc, ) print_error( - f"Merge [bold]{merge_id}[/bold] failed because the " - f"shared {backend_display_name(config.agent.backend)} " - f"credential could not be used or refreshed — this " - f"is a login failure, not a failure of the merged " - f"code. Falling through to mutation." + f"Merge [bold]{merge_id}[/bold] " + + _credential_failure_verdict( + config.agent.backend, + transient=_merge_cred_exc.transient, + subject="the merged code", + ) + + " Falling through to mutation." ) if merged is None: @@ -2596,12 +2741,17 @@ def _has_val_support_overlap(i: str, j: str) -> bool: candidate_id=merge_id, source="merge_failed", ) - print_error( - f"Merge {merge_id} failed " - f"(candidates: {a.id} + {b.id}, gen {gen}). " - f"Claude Code returned no output or the merge subprocess errored. " - f"Check the HELIX ERROR panel above for full diagnostics." - ) + # A credential failure has already been diagnosed + # above; the generic wording would contradict it. + if not merge_credential_failed: + print_error( + f"Merge {merge_id} failed " + f"(candidates: {a.id} + {b.id}, gen {gen}). " + f"{backend_display_name(config.agent.backend)} " + "returned no output or the merge subprocess " + "errored. Check the HELIX ERROR panel above " + "for full diagnostics." + ) else: if merged.usage: live.update(usage=merged.usage) @@ -3653,34 +3803,45 @@ def _drop_duplicate_child(gated: GatedProposal) -> bool: # able to tell them apart after the fact. The per-slot errors above have # long scrolled away by now; this line is part of the permanent summary # that outlives the live display. + _display = backend_display_name(config.agent.backend) + if credential_failures.recovered_ids(): + # Not failures -- but the only visible sign that candidates are + # still racing to refresh the shared login despite the warm. + _recovered_ids = ", ".join(credential_failures.recovered_ids()) + print_warning( + f"{len(credential_failures.recovered_ids())} invocation(s) " + f"recovered after a lost refresh race on the shared {_display} " + f"credential: {_recovered_ids}. Each was retried once from a " + "fresh worktree and succeeded; both attempts' tokens are charged " + "and the first attempt's output is kept beside the retry's " + "(`.attempt1` artifacts). The credential is being refreshed by " + "candidates in flight, which the per-generation warm does not " + "prevent." + ) if credential_failures: _failed_ids = ", ".join(credential_failures.candidate_ids()) - _display = backend_display_name(config.agent.backend) - if credential_failures.all_transient(): + _transient = credential_failures.all_transient() + if _transient: # Every failure was a lost refresh race: the shared login was # refreshed by another candidate and is most likely fine. Telling # the operator to re-login here would throw away a working # credential and teach them to distrust a healthy run. - print_error( - f"{len(credential_failures)} mutation(s) failed on the shared " - f"{_display} credential, not on their code: {_failed_ids}. " - f"Each lost a refresh race (another candidate refreshed the " - f"shared login first) and failed again on its one retry. The " - f"stored login is most likely usable: run " - f"[cyan]helix resume[/cyan] first, and only re-authenticate " - f"if this keeps recurring. Last report: " - f"{credential_failures.last_message()}" + _cause = ( + "Each lost a refresh race (another candidate refreshed the " + "shared login first) and failed again on its one retry." ) else: - print_error( - f"{len(credential_failures)} mutation(s) failed on the shared " - f"{_display} credential, not on their code: {_failed_ids}. " - f"The backend reported that its stored login could not be " - f"used or refreshed. Re-authenticate with " - f"[cyan]helix sandbox login {config.agent.backend}[/cyan], " - f"then [cyan]helix resume[/cyan]. Last report: " - f"{credential_failures.last_message()}" + _cause = ( + "The backend reported that its stored login could not be " + "used or refreshed." ) + print_error( + f"{credential_failures.describe_failed()} failed on the shared " + f"{_display} credential, not on their code: {_failed_ids}. " + f"{_cause} " + + _credential_remedy(config.agent.backend, transient=_transient) + + f" Last report: {escape(credential_failures.last_message())}" + ) best = frontier.best() diff --git a/src/helix/state.py b/src/helix/state.py index 5293915e..97bc4df3 100644 --- a/src/helix/state.py +++ b/src/helix/state.py @@ -156,6 +156,11 @@ def _eval_cache_path(base_dir: Path) -> Path: return base_dir / _STATE_DIR / _EVAL_CACHE_FILENAME +def state_file_exists(base_dir: Path) -> bool: + """True when ``/.helix/state.json`` has been written.""" + return _state_path(base_dir).is_file() + + def save_state(state: EvolutionState, base_dir: Path) -> None: """Atomically write the evolution state to .helix/state.json.""" target = _state_path(base_dir) diff --git a/tests/unit/test_cli_credential_failure.py b/tests/unit/test_cli_credential_failure.py index 7da39d85..110ddd60 100644 --- a/tests/unit/test_cli_credential_failure.py +++ b/tests/unit/test_cli_credential_failure.py @@ -25,10 +25,14 @@ def _make_project(tmp_path: Path) -> Path: @pytest.mark.parametrize("command", ["evolve", "resume"]) +@pytest.mark.parametrize("state_saved", [True, False]) def test_escaped_credential_error_is_a_panel_not_a_traceback( - mocker, tmp_path: Path, command: str + mocker, tmp_path: Path, command: str, state_saved: bool ) -> None: project = _make_project(tmp_path) + if state_saved: + (project / ".helix").mkdir() + (project / ".helix" / "state.json").write_text("{}") # Both commands import run_evolution lazily inside the function body. mocker.patch( "helix.evolution.run_evolution", @@ -48,4 +52,29 @@ def test_escaped_credential_error_is_a_panel_not_a_traceback( out = " ".join(result.output.lower().split()) assert "credential" in out assert "helix sandbox login" in out + if state_saved: + assert "state has been saved" in out + assert "helix resume" in out + else: + # The only path that reaches this handler is seedless seed + # generation, which fails before the first save; ``helix resume`` + # would start a fresh run and repeat the failure. + assert "no evolution state was saved" in out + assert "helix resume" not in out + assert "helix evolve" in out + + +def test_transient_failure_does_not_demand_a_relogin(mocker, tmp_path: Path) -> None: + project = _make_project(tmp_path) + (project / ".helix").mkdir() + (project / ".helix" / "state.json").write_text("{}") + exc = CredentialRefreshError("lost a refresh race", suggestion="resume") + exc.transient = True + mocker.patch("helix.evolution.run_evolution", side_effect=exc) + + result = CliRunner().invoke(cli, ["evolve", "--dir", str(project)]) + + out = " ".join(result.output.lower().split()) + assert result.exit_code == 2 assert "helix resume" in out + assert "helix sandbox login" not in out diff --git a/tests/unit/test_credential_warm.py b/tests/unit/test_credential_warm.py index 04828e90..8ebee4c8 100644 --- a/tests/unit/test_credential_warm.py +++ b/tests/unit/test_credential_warm.py @@ -516,15 +516,105 @@ def test_timed_out_warm_is_a_warning_in_the_loop( # --------------------------------------------------------------------------- -def _config(backend: str, *, sandboxed: bool) -> HelixConfig: +def _config( + backend: str, + *, + sandboxed: bool, + num_parallel_proposals: int = 2, + mutations_per_parent: int = 1, + max_workers: int = 4, +) -> HelixConfig: return HelixConfig( objective="Improve the code", evaluator=EvaluatorConfig(command="pytest -q"), agent=AgentConfig(backend=backend), # type: ignore[arg-type] sandbox=SandboxConfig(enabled=sandboxed), + evolution=EvolutionConfig( + num_parallel_proposals=num_parallel_proposals, + mutations_per_parent=mutations_per_parent, + max_workers=max_workers, + ), ) +class TestWarmIsGatedOnConcurrency: + """A single writer cannot lose a refresh race to itself.""" + + @pytest.mark.parametrize( + "kwargs", + [ + dict(num_parallel_proposals=1, mutations_per_parent=1, max_workers=8), + dict(num_parallel_proposals=4, mutations_per_parent=1, max_workers=1), + dict(num_parallel_proposals=1, mutations_per_parent=1, max_workers=1), + ], + ) + def test_single_writer_skips_the_warm_and_says_why( + self, + monkeypatch: pytest.MonkeyPatch, + caplog: pytest.LogCaptureFixture, + kwargs: dict[str, int], + ) -> None: + def _boom(*_a: Any, **_k: Any) -> None: + raise AssertionError("no warm with a single writer") + + monkeypatch.setattr("helix.evolution.warm_backend_credential", _boom) + with caplog.at_level("INFO", logger="helix.evolution"): + result = _warm_generation_credential( + _config("codex", sandboxed=True, **kwargs), gen=1, announce_skip=True + ) + assert result is not None and result.skipped + assert "single writer" in (result.skip_reason or "") + assert "single writer" in caplog.text + + @pytest.mark.parametrize( + "kwargs", + [ + dict(num_parallel_proposals=2, mutations_per_parent=1, max_workers=8), + dict(num_parallel_proposals=1, mutations_per_parent=2, max_workers=2), + ], + ) + def test_two_writers_warm( + self, monkeypatch: pytest.MonkeyPatch, kwargs: dict[str, int] + ) -> None: + calls: list[str] = [] + monkeypatch.setattr( + "helix.evolution.warm_backend_credential", + lambda backend, **_k: ( + calls.append(backend), + CredentialWarmResult(backend=backend, warmed=True, returncode=0), + )[1], + ) + _warm_generation_credential( + _config("codex", sandboxed=True, **kwargs), gen=1, announce_skip=True + ) + assert calls == ["codex"] + + def test_operator_env_is_passed_to_the_warm( + self, monkeypatch: pytest.MonkeyPatch + ) -> None: + seen: dict[str, Any] = {} + monkeypatch.setattr( + "helix.evolution.warm_backend_credential", + lambda backend, **k: ( + seen.update(k), + CredentialWarmResult(backend=backend, warmed=True, returncode=0), + )[1], + ) + monkeypatch.setenv("HTTPS_PROXY", "http://proxy:3128") + monkeypatch.delenv("NOT_SET", raising=False) + config = _config("codex", sandboxed=True).model_copy( + update={ + "passthrough_env": ["HTTPS_PROXY", "NOT_SET"], + "env": {"SSL_CERT_FILE": "/ca.pem"}, + } + ) + _warm_generation_credential(config, gen=1, announce_skip=True) + assert seen["env"] == { + "HTTPS_PROXY": "http://proxy:3128", + "SSL_CERT_FILE": "/ca.pem", + } + + class TestGenerationWarm: def test_unsandboxed_run_warms_nothing( self, monkeypatch: pytest.MonkeyPatch @@ -576,3 +666,24 @@ def test_failed_warm_returns_and_does_not_raise( printed = capsys.readouterr().out assert "refresh" in printed.lower() assert "run continues" in printed.lower() + + def test_container_output_with_rich_markup_does_not_crash_the_run( + self, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] + ) -> None: + """A stderr tail like ``could not read [/home/node/.codex/auth.json]`` + or a truncated ``[/bold]`` raised ``rich.errors.MarkupError`` on the + main thread before any candidate was dispatched.""" + detail = "could not read [/home/node/.codex/auth.json] [/] [/bold] [auth]" + monkeypatch.setattr( + "helix.evolution.warm_backend_credential", + lambda backend, **_k: CredentialWarmResult( + backend=backend, warmed=False, returncode=1, detail=detail + ), + ) + result = _warm_generation_credential( + _config("codex", sandboxed=True), gen=2, announce_skip=False + ) + assert result is not None and result.failed + out = capsys.readouterr().out + assert "[/bold]" in out + assert "[auth]" in out diff --git a/tests/unit/test_credential_warm_loop.py b/tests/unit/test_credential_warm_loop.py index 829e389d..2bd111d9 100644 --- a/tests/unit/test_credential_warm_loop.py +++ b/tests/unit/test_credential_warm_loop.py @@ -17,7 +17,7 @@ import pytest -from helix.config import SandboxConfig +from helix.config import AgentConfig, SandboxConfig from helix.evolution import run_evolution from helix.exceptions import CredentialRefreshError from helix.sandbox import CredentialWarmResult @@ -30,7 +30,21 @@ def _sandboxed(config: Any) -> Any: - return config.model_copy(update={"sandbox": SandboxConfig(enabled=True)}) + """Sandboxed, on the one backend the code actually warms (codex).""" + return config.model_copy( + update={ + "sandbox": SandboxConfig(enabled=True), + "agent": AgentConfig(backend="codex"), + } + ) + + +@pytest.fixture(autouse=True) +def two_writers(mocker: Any) -> None: + """The warm is skipped for a single writer; that gate has its own tests + (``test_credential_warm.py``). Here the loop shape is what matters, so + pretend every generation has two concurrent writers.""" + mocker.patch("helix.evolution._generation_concurrent_writers", return_value=2) @pytest.fixture() @@ -69,7 +83,7 @@ def test_one_warm_per_generation( ) run_evolution(config, tmp_path, tmp_path / ".helix") - assert warm_calls == ["claude", "claude", "claude"] + assert warm_calls == ["codex", "codex", "codex"] def test_warm_precedes_every_mutation( self, mocker, tmp_path, all_mocks # noqa: F811 @@ -218,10 +232,60 @@ def run_eval(candidate, config, split=None, instances=None, **kwargs): assert "merge" in out assert "credential" in out assert "not a failure of the merged code" in out - # The end-of-run summary names the merge slot, not just the mutation. - assert "1 mutation(s) failed on the shared" in out + # Exactly one diagnosis at the merge site: the generic "returned no + # output" wording must not follow and contradict it. + assert out.count("not a failure of the merged code") == 1 + assert "returned no output" not in out + # The end-of-run summary labels the merge slot as a merge. + assert "1 merge(s) failed on the shared" in out + assert "mutation(s) failed" not in out assert "helix sandbox login" in out + def test_merge_lost_refresh_race_is_worded_as_transient( + self, + mocker, # noqa: F811 + tmp_path, + all_mocks, # noqa: F811 + warm_calls, + capsys: pytest.CaptureFixture[str], + ) -> None: + seed = make_candidate("g0-s0") + child = make_candidate("g1-s1", generation=1) + all_mocks["create_seed_worktree"].return_value = seed + all_mocks["mutate"].return_value = child + exc = CredentialRefreshError( + "Codex CLI lost a refresh race on the shared credential " + "(matched 'because your refresh token was already used' in " + "stderr; failed again on retry)", + suggestion="Run `helix resume`.", + ) + exc.transient = True + all_mocks["merge"].side_effect = exc + all_mocks["find_merge_triplet"].return_value = ("g0-s0", "g1-s1", "g0-s0") + + def run_eval(candidate, config, split=None, instances=None, **kwargs): + if candidate.id == "g1-s1": + return make_eval_result("g1-s1", {"i1": 0.9, "i2": 0.5}) + return make_eval_result(candidate.id, {"i1": 0.5, "i2": 0.8}) + + all_mocks["run_evaluator"].side_effect = run_eval + config = _sandboxed( + make_config( + max_generations=2, + merge_enabled=True, + max_merge_invocations=5, + merge_val_overlap_floor=1, + max_evaluations=10000, + ) + ) + run_evolution(config, tmp_path, tmp_path / ".helix") + + out = " ".join(capsys.readouterr().out.lower().split()) + assert "refreshed by another candidate first" in out + assert "could not be used or refreshed" not in out + assert "helix sandbox login" not in out + assert "helix resume" in out + def test_lost_refresh_race_does_not_demand_a_relogin( self, mocker, # noqa: F811 @@ -283,3 +347,45 @@ def test_clean_run_says_nothing_about_credentials( run_evolution(config, tmp_path, tmp_path / ".helix") assert "credential" not in capsys.readouterr().out.lower() + + def test_recovered_refresh_race_reaches_the_summary( + self, + mocker, # noqa: F811 + tmp_path, + all_mocks, # noqa: F811 + warm_calls, + capsys: pytest.CaptureFixture[str], + ) -> None: + """A race the retry recovered from is not a failure, but it is the + only sign the operator gets that candidates are still refreshing the + shared login for themselves.""" + seed = make_candidate("g0-s0") + all_mocks["create_seed_worktree"].return_value = seed + all_mocks["run_evaluator"].side_effect = ( + lambda candidate, *a, **k: make_eval_result( + candidate.id, {"i1": 0.5, "i2": 0.5} + ) + ) + + def _mutate(*args: Any, **kwargs: Any) -> None: + kwargs["on_refresh_race_recovered"]( + f"{kwargs['new_id']} recovered after a lost refresh race on the " + "shared Codex CLI credential" + ) + return None + + all_mocks["mutate"].side_effect = _mutate + config = _sandboxed( + make_config(max_generations=2, perfect_score_threshold=None) + ) + run_evolution(config, tmp_path, tmp_path / ".helix") + + assert all_mocks["mutate"].call_count >= 1 + # ``all_mocks`` stubs ``print_warning``; the summary goes through it + # because a recovered race is a warning, not a failure. + warnings = " ".join( + str(call.args[0]) for call in all_mocks["print_warning"].call_args_list + ).lower() + assert "recovered after a lost refresh race" in warnings + assert "g1-s1" in warnings + assert "failed on the shared" not in capsys.readouterr().out.lower() diff --git a/tests/unit/test_evolution_seedless.py b/tests/unit/test_evolution_seedless.py index 72cdcfd5..6db015c1 100644 --- a/tests/unit/test_evolution_seedless.py +++ b/tests/unit/test_evolution_seedless.py @@ -220,6 +220,29 @@ def test_raises_immediately_when_generate_seed_fails(self, tmp_path, seedless_mo assert exc_info.value is exc + def test_failed_seed_generation_still_charges_its_usage( + self, tmp_path, seedless_mocks, mocker + ): + """A seed invocation that dies on the shared credential spent tokens + first; they are charged before the worktree goes and the error + propagates, mirroring ``merge()`` / ``mutate()``.""" + from helix.exceptions import CredentialRefreshError + + config = make_config(seedless=True) + usage = UsageStats(input_tokens=9, output_tokens=3) + seedless_mocks["generate_seed"].side_effect = CredentialRefreshError( + "login is dead", usage=usage + ) + charge = mocker.patch("helix.evolution.budget_api.charge_llm_usage") + + with pytest.raises(CredentialRefreshError): + run_evolution(config, tmp_path, tmp_path / ".helix") + + [call] = [c for c in charge.call_args_list if c.args[1] is usage] + assert call.kwargs["candidate_id"] == "g0-s0" + assert call.kwargs["source"] == "seed_generation_failed" + seedless_mocks["remove_worktree"].assert_called_once() + def test_generate_seed_called_only_once_on_failure(self, tmp_path, seedless_mocks): """generate_seed must only be called once even on failure (no retry).""" config = make_config(seedless=True) From 587b7586ab21d9d36d6e313732bf6296b7d6807f Mon Sep 17 00:00:00 2001 From: Karim Elmaaroufi Date: Thu, 10 Sep 2026 21:10:13 -0700 Subject: [PATCH 16/16] docs: soften the fresh-session and warm claims The two switches stop the CLIs' own cross-session memory -- the only spontaneous channel the probe measured. They are not a session-isolation guarantee: a candidate with shell access can still write the shared HOME's config files (~/.claude/CLAUDE.md, ~/.claude/settings.json, ~/.codex/AGENTS.md, ~/.codex/config.toml), which later sessions load; that channel is deliberately left open because the volume is shared by design. BACKEND_FRESH_SESSION_ENV applies to unsandboxed runs too (the CHANGELOG scoped it to sandboxed ones) and an operator's explicit [env] value for the key wins. README and CHANGELOG describe the warm as making the credential fresh at the start of a generation, verified by reading last_refresh back, with lost races retried once and reported. Co-Authored-By: Claude Fable 5.1 Claude-Session: https://claude.ai/code/session_0126UoDvKj2BN5SHLH81aqnW --- CHANGELOG.md | 20 ++++++++++++++++---- README.md | 21 +++++++++++++++++---- src/helix/backends.py | 14 ++++++++++++-- 3 files changed, 45 insertions(+), 10 deletions(-) diff --git a/CHANGELOG.md b/CHANGELOG.md index 4644528d..7bc556fc 100644 --- a/CHANGELOG.md +++ b/CHANGELOG.md @@ -8,10 +8,22 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0 ## [Unreleased] ### Added -- Every sandboxed candidate now starts with a fresh agent session: - `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` for `claude` and - `-c features.memories=false` for `codex` (the other backends read nothing - from a prior session); transcripts stay in the `helix-auth-` volume. +- The agent CLIs' own cross-session memory is switched off for every + candidate, sandboxed or not: `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` for `claude` + and `-c features.memories=false` for `codex` (the other backends recalled + nothing from a prior session on their own). An explicit `[env]` value for + the same key wins. This is not a session-isolation guarantee: the login + volume is shared by design, and a candidate with shell access can still + write the shared HOME's config files (`~/.claude/CLAUDE.md`, + `~/.claude/settings.json`, `~/.codex/AGENTS.md`, `~/.codex/config.toml`), + which later sessions load. Transcripts stay in the `helix-auth-` + volume. +- Sandboxed runs with more than one concurrent candidate warm the shared + `codex` credential once per generation under a single writer, verified by + reading `last_refresh` back from `auth.json`; a credential failure is now + its own error kind (`CredentialRefreshError`), a lost refresh race is + retried once from a fresh worktree, and both are named in the end-of-run + summary. ### Changed - **BREAKING**: Removed the `gemini` mutation backend and replaced it with diff --git a/README.md b/README.md index b0568ff9..28298f45 100644 --- a/README.md +++ b/README.md @@ -645,13 +645,26 @@ evaluator uses a local proxy, keep that endpoint in your evaluator code as usual. Docker Desktop supports `host.docker.internal`; Linux users can set `add_host_gateway = true`. -Every candidate starts with a fresh agent session: HELIX sets +HELIX stops the agent CLIs' own cross-session memory: it sets `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` for `claude` (its auto-memory is keyed by repo root and would otherwise span candidates) and passes `-c features.memories=false` to `codex`; `agy`, `cursor`, and `opencode` were -probed and read nothing from a prior session, so they need no switch. -Transcripts and session databases remain in the `helix-auth-` volume, -so operators can read them after a run. +probed and recalled nothing from a prior session on their own, so they need no +switch. Both switches apply to sandboxed and unsandboxed runs, and an explicit +`[env]` value for the same key wins. This is not a session-isolation guarantee: +the login volume is shared by design, so a candidate with shell access can +still write the shared HOME's config files (`~/.claude/CLAUDE.md`, +`~/.claude/settings.json`, `~/.codex/AGENTS.md`, `~/.codex/config.toml`), +which later sessions load. Transcripts and session databases remain in the +`helix-auth-` volume, so operators can read them after a run. + +When several candidates can write the shared login at once, HELIX refreshes +the `codex` credential once per generation under a single writer before +dispatching anything (`codex debug models`, verified by reading `last_refresh` +back from `auth.json`), so the credential is fresh at the start of the +generation. A token that crosses its refresh threshold during the generation +can still be raced by candidates in flight; a lost race is retried once from +a fresh worktree and reported in the end-of-run summary. The container-backed tests in `tests/integration/` run real backend images, so a bare `pytest` does not collect them; opt in with diff --git a/src/helix/backends.py b/src/helix/backends.py index f41dc897..a07806dd 100644 --- a/src/helix/backends.py +++ b/src/helix/backends.py @@ -73,13 +73,23 @@ "opencode": ("OPENCODE_API_KEY", "OPENAI_API_KEY", "ANTHROPIC_API_KEY"), } -# Environment that makes every candidate start from a fresh agent session. +# Environment that stops the agent CLIs' own cross-session memory. # # Candidates share one login volume per backend (mounted at /home/node), so a # CLI that reads memory from an earlier session would carry state from # candidate N-1 into candidate N. Probed 2026-09-10 against the real CLIs: # plant a fact in one one-shot session, ask for it in a fresh one, same cwd, -# no tools. +# no tools. That probe measures the spontaneous channel only -- what a CLI +# recalls on its own. It is not a session-isolation guarantee: a candidate +# with shell access can still write the shared HOME's config files +# (``~/.claude/CLAUDE.md``, ``~/.claude/settings.json``, ``~/.codex/AGENTS.md``, +# ``~/.codex/config.toml``), which later sessions load. That channel is +# deliberately left open, because the volume is shared by design so that +# transcripts and a refreshed login persist across candidates. +# +# Applies to sandboxed and unsandboxed runs alike (``invoke_claude_code`` +# sets it on the backend environment either way). An operator who names the +# same key in ``[env]`` wins: the value here is a default, not an override. # claude recalled it -- auto-memory, keyed by repo root, so it spans # worktrees and the shared HOME; CLAUDE_CODE_DISABLE_AUTO_MEMORY=1 # stops it and no memory directory is created.