feat(sandbox): warm the shared credential once per generation, name credential failures, and start every candidate fresh

CHANGELOG.md

@@ -7,6 +7,24 @@ and this project adheres to [Semantic Versioning](https://semver.org/spec/v2.0.0
 
 ## [Unreleased]
 
+### Added
+- The agent CLIs' own cross-session memory is switched off for every
+  candidate, sandboxed or not: `CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` for `claude`
+  and `-c features.memories=false` for `codex` (the other backends recalled
+  nothing from a prior session on their own). An explicit `[env]` value for
+  the same key wins. This is not a session-isolation guarantee: the login
+  volume is shared by design, and a candidate with shell access can still
+  write the shared HOME's config files (`~/.claude/CLAUDE.md`,
+  `~/.claude/settings.json`, `~/.codex/AGENTS.md`, `~/.codex/config.toml`),
+  which later sessions load. Transcripts stay in the `helix-auth-<backend>`
+  volume.
+- Sandboxed runs with more than one concurrent candidate warm the shared
+  `codex` credential once per generation under a single writer, verified by
+  reading `last_refresh` back from `auth.json`; a credential failure is now
+  its own error kind (`CredentialRefreshError`), a lost refresh race is
+  retried once from a fresh worktree, and both are named in the end-of-run
+  summary.
+
 ### Changed
 - **BREAKING**: Removed the `gemini` mutation backend and replaced it with
   `agy` (Google's Antigravity CLI). Configs with `agent.backend = "gemini"`

README.md

@@ -645,6 +645,33 @@ evaluator uses a local proxy, keep that endpoint in your evaluator code as
 usual. Docker Desktop supports `host.docker.internal`; Linux users can set
 `add_host_gateway = true`.
 
+HELIX stops the agent CLIs' own cross-session memory: it sets
+`CLAUDE_CODE_DISABLE_AUTO_MEMORY=1` for `claude` (its auto-memory is keyed by
+repo root and would otherwise span candidates) and passes
+`-c features.memories=false` to `codex`; `agy`, `cursor`, and `opencode` were
+probed and recalled nothing from a prior session on their own, so they need no
+switch. Both switches apply to sandboxed and unsandboxed runs, and an explicit
+`[env]` value for the same key wins. This is not a session-isolation guarantee:
+the login volume is shared by design, so a candidate with shell access can
+still write the shared HOME's config files (`~/.claude/CLAUDE.md`,
+`~/.claude/settings.json`, `~/.codex/AGENTS.md`, `~/.codex/config.toml`),
+which later sessions load. Transcripts and session databases remain in the
+`helix-auth-<backend>` volume, so operators can read them after a run.
+
+When several candidates can write the shared login at once, HELIX refreshes
+the `codex` credential once per generation under a single writer before
+dispatching anything (`codex debug models`, verified by reading `last_refresh`
+back from `auth.json`), so the credential is fresh at the start of the
+generation. A token that crosses its refresh threshold during the generation
+can still be raced by candidates in flight; a lost race is retried once from
+a fresh worktree and reported in the end-of-run summary.
+
+The container-backed tests in `tests/integration/` run real backend images,
+so a bare `pytest` does not collect them; opt in with
+`pytest -m docker_integration tests/integration/` (set
+`HELIX_DOCKER_TESTS_STRICT=1` to fail rather than skip when Docker or an image
+is missing).
+
 By default HELIX chooses a published backend-specific mutator image from
 `agent.backend`: `ghcr.io/ke7/helix-evo-runner-agy`,
 `ghcr.io/ke7/helix-evo-runner-claude`,

pyproject.toml

@@ -81,6 +81,12 @@ testpaths = ["tests"]
 # (a now-fixed NB-2-style infinite loop that wedged CI for 24 minutes).
 timeout = 60
 timeout_method = "thread"
+# tests/integration/ runs real backend containers and is not run in CI.  Keep
+# a bare `pytest` from collecting it by default; opt in with
+# `pytest -m docker_integration tests/integration/` (a command-line -m
+# overrides this).
+addopts = "-m 'not docker_integration'"
 markers = [
     "diff_harness: differential-testing harness (phases 2-4)",
+    "docker_integration: runs real containers; needs a Docker daemon and the backend images",
 ]

src/helix/backends.py

@@ -73,6 +73,43 @@
     "opencode": ("OPENCODE_API_KEY", "OPENAI_API_KEY", "ANTHROPIC_API_KEY"),
 }
 
+# Environment that stops the agent CLIs' own cross-session memory.
+#
+# Candidates share one login volume per backend (mounted at /home/node), so a
+# CLI that reads memory from an earlier session would carry state from
+# candidate N-1 into candidate N.  Probed 2026-09-10 against the real CLIs:
+# plant a fact in one one-shot session, ask for it in a fresh one, same cwd,
+# no tools.  That probe measures the spontaneous channel only -- what a CLI
+# recalls on its own.  It is not a session-isolation guarantee: a candidate
+# with shell access can still write the shared HOME's config files
+# (``~/.claude/CLAUDE.md``, ``~/.claude/settings.json``, ``~/.codex/AGENTS.md``,
+# ``~/.codex/config.toml``), which later sessions load.  That channel is
+# deliberately left open, because the volume is shared by design so that
+# transcripts and a refreshed login persist across candidates.
+#
+# Applies to sandboxed and unsandboxed runs alike (``invoke_claude_code``
+# sets it on the backend environment either way).  An operator who names the
+# same key in ``[env]`` wins: the value here is a default, not an override.
+#   claude   recalled it -- auto-memory, keyed by repo root, so it spans
+#            worktrees and the shared HOME; CLAUDE_CODE_DISABLE_AUTO_MEMORY=1
+#            stops it and no memory directory is created.
+#   codex    did not; its ``memories`` feature flag is off by default and is
+#            pinned off in argv (``-c features.memories=false``, see
+#            ``helix.mutator._build_backend_args``) because it is a config
+#            override, not an environment variable.
+#   agy, cursor, opencode   did not; they read nothing from a prior session,
+#            so there is nothing to disable and no entry here.
+# Transcripts and session databases are still written to the login volume so
+# they can be read after a run.
+BACKEND_FRESH_SESSION_ENV: dict[str, dict[str, str]] = {
+    "claude": {"CLAUDE_CODE_DISABLE_AUTO_MEMORY": "1"},
+}
+
+# Every shell entry below is ``sh -c``, never ``sh -lc``: a login shell sources
+# ``/etc/profile`` and ``$HOME/.profile`` from the shared login volume, which
+# every candidate container mounts read-write, so ``-l`` would let a candidate
+# plant code that runs in the next auth command.  PATH is pinned with ``-e`` by
+# ``helix.sandbox.sandbox_auth_docker_args``; nothing here needs a profile.
 BACKEND_AUTH_COMMANDS: dict[str, dict[str, list[str]]] = {
     "agy": {
         # No dedicated non-interactive login subcommand; the bare interactive
@@ -85,15 +122,15 @@
         # confirmed against a completed sign-in.
         "status": [
             "sh",
-            "-lc",
+            "-c",
             'set -eu; test -s "${HOME:-/home/node}/.gemini/antigravity-cli/antigravity-oauth-token"',
         ],
         # Surgical: only remove agy's own state directory. ``~/.gemini`` also
         # holds unrelated Google CLI state, so a blanket ``rm -rf ~/.gemini``
         # would destroy state this backend does not own.
         "logout": [
             "sh",
-            "-lc",
+            "-c",
             'set -eu; rm -rf "${HOME:-/home/node}/.gemini/antigravity-cli"',
         ],
     },
@@ -106,7 +143,7 @@
         # localised in some CLI versions).
         "status": [
             "sh",
-            "-lc",
+            "-c",
             "set -eu; "
             "claude auth status --text 2>&1 || true; "
             'test -s "${HOME:-/home/node}/.claude/.credentials.json"',
@@ -117,6 +154,38 @@
         "login": ["codex", "login", "--device-auth"],
         "status": ["codex", "login", "status"],
         "logout": ["codex", "logout"],
+        # Credential warm -- see CREDENTIAL_WARM_SKIP_REASONS below for why
+        # codex is the only backend with one.
+        #
+        # NOT ``codex login status``.  Measured against codex-cli 0.130.0 with
+        # a synthetic credential in a throwaway volume: ``codex login status``
+        # prints "Logged in using ChatGPT" and exits 0 without issuing a single
+        # request, whether the stored ``last_refresh`` is minutes or 30 days
+        # old.  It reads auth.json; it never takes the refresh path, so warming
+        # with it would be a placebo.
+        #
+        # ``codex debug models`` renders the CLI's model catalog.  It loads
+        # auth through the refreshing path, so it performs the refresh this
+        # warm exists to perform, and it is free:
+        #   * with a fresh credential it completes with ``--network none`` and
+        #     makes no request at all -- the only thing it may write to the
+        #     login volume is its own ``~/.codex/models_cache.json`` (a
+        #     catalog cache with a 5-minute TTL), never the credential;
+        #   * with a stale credential its only request is the OAuth token
+        #     exchange, which the refreshed credential is then written back
+        #     from.  No model is invoked and no quota is consumed either way.
+        # Its exit code says nothing about the refresh: a rejected exchange is
+        # logged and swallowed and the command still exits 0.  That is why
+        # ``helix.sandbox.warm_backend_credential`` reads ``last_refresh``
+        # back from ``auth.json`` and only reports ``warmed`` when the
+        # credential is verifiably inside codex's refresh interval.  There is
+        # no flag to bypass the catalog cache (``--bundled`` does the
+        # opposite: it skips the refresh), so a cache younger than 5 minutes
+        # can short-circuit the refresh; the read-back catches that too.
+        # stdout is discarded because the catalog is ~200 KB and the command is
+        # run for its side effect on the credential, not for its output;
+        # stderr is kept so a failure stays diagnosable.
+        "warm": ["sh", "-c", "set -eu; codex debug models >/dev/null"],
     },
     "cursor": {
         "login": ["cursor-agent", "login"],
@@ -131,5 +200,87 @@
 }
 
 
+# ---------------------------------------------------------------------------
+# Per-generation credential warm
+# ---------------------------------------------------------------------------
+#
+# Every candidate container mounts the shared login volume read-write, which is
+# what lets a backend CLI refresh its own OAuth token and keep the refreshed
+# credential for the next candidate.  The hazard is the *first* moment after a
+# credential goes stale: several candidates start at once, each decides
+# independently that a refresh is due, and each posts the same single-use
+# refresh token.  One wins; the rest are told the token was already consumed.
+#
+# ``helix.sandbox.warm_backend_credential`` narrows that window by running the
+# command below once, in one container, before a generation dispatches any
+# candidate -- so a refresh that is due at that moment happens under a single
+# writer.  The credential is then fresh at the *start* of the generation; a
+# token that crosses its refresh threshold during the generation (parent
+# evaluations run before each mutation, and queued slots start later still)
+# can still be raced by the candidates in flight.  The warm is skipped when
+# at most one candidate can write the shared login at a time, since a single
+# writer cannot race itself.
+#
+# A backend is warmed only when a command exists here that (a) actually takes
+# the CLI's refresh path and (b) costs nothing.  Both halves are load-bearing:
+# a command that never refreshes buys no safety, and a command that bills the
+# operator's account once per generation would be worse than the race it
+# prevents.  Backends with no entry are listed in
+# CREDENTIAL_WARM_SKIP_REASONS with the reason they need none.
+
+
+CREDENTIAL_WARM_SKIP_REASONS: dict[str, str] = {
+    "claude": (
+        "Claude Code serialises its own refresh: it takes a real cross-process "
+        "lock file, retries while another process holds it, and re-reads the "
+        "credential afterwards, so concurrent candidates cannot consume the "
+        "same refresh token. Warming would add a container per generation and "
+        "remove no hazard."
+    ),
+    "cursor": (
+        "Cursor Agent never spends its stored refresh token: it re-exchanges "
+        "an API key instead, so there is no single-use grant for candidates to "
+        "compete over."
+    ),
+    "agy": (
+        "No free Antigravity CLI command is known to take the refresh path. "
+        "The registered status probe is a file test "
+        "(`test -s ~/.gemini/antigravity-cli/antigravity-oauth-token`), which "
+        "touches no credential path, so warming with it would be a placebo. "
+        "`agy models` exits 0 even when logged out, and whether it takes the "
+        "refresh path has not been measured; no agy credential exists to "
+        "measure one against. Left unwarmed deliberately rather than warmed "
+        "on a guess."
+    ),
+    "opencode": (
+        "OpenCode refreshes an `oauth`-type credential only from inside the "
+        "fetch wrapper that issues a model request -- read from opencode-ai "
+        "1.14.24, which refreshes when `expires` has passed and writes the new "
+        "credential back unlocked. There is therefore no command that performs "
+        "that refresh without also invoking a model, and a per-generation model "
+        "call on the operator's account is a worse cost than the race. "
+        "`opencode providers list` was measured to be free -- it completes with "
+        "`--network none` against an expired oauth credential and leaves "
+        "auth.json byte-identical -- but for exactly that reason it refreshes "
+        "nothing. `api`-type credentials never refresh and are not at risk."
+    ),
+}
+"""Why a backend has no ``warm`` entry in :data:`BACKEND_AUTH_COMMANDS`.
+
+Skipping is a correctness statement, not an optimisation: each entry records
+either that the backend cannot lose a refresh race, or that no free command
+would win it.
+"""
+
+
+def backend_credential_warm_skip_reason(backend: str) -> str | None:
+    """Return why *backend* is not credential-warmed, or ``None`` if it is."""
+    if "warm" in BACKEND_AUTH_COMMANDS.get(backend, {}):
+        return None
+    return CREDENTIAL_WARM_SKIP_REASONS.get(
+        backend, "no credential-warm command is registered for this backend"
+    )
+
+
 def backend_display_name(backend: str) -> str:
     return BACKEND_DISPLAY_NAMES.get(backend, backend)

src/helix/cli.py

@@ -26,10 +26,15 @@
     print_warning,
     render_frontier_table,
 )
-from helix.exceptions import RateLimitError, ResumeIncompatibleError, print_helix_error
+from helix.exceptions import (
+    CredentialRefreshError,
+    RateLimitError,
+    ResumeIncompatibleError,
+    print_helix_error,
+)
 from helix.lineage import load_lineage
 from helix.population import EvalResult, FrontierType, ParetoFrontier, Candidate
-from helix.state import load_state, save_state
+from helix.state import load_state, save_state, state_file_exists
 from helix.worktree import remove_worktree
 
 logger = logging.getLogger(__name__)
@@ -109,6 +114,28 @@ def _helix_dir(project_root: Path) -> Path:
     return project_root / _HELIX_DIR
 
 
+def _print_credential_failure_hint(
+    project_root: Path, backend: str, exc: CredentialRefreshError
+) -> None:
+    """Say what to do after a credential failure -- truthfully about state.
+
+    Seedless seed generation is the one path that reaches the CLI handler,
+    and it fails before the first ``save_state``; promising ``helix resume``
+    then sends the operator to a command that starts a fresh run.
+    """
+    from helix.evolution import _credential_remedy
+
+    remedy = _credential_remedy(backend, transient=exc.transient)
+    if state_file_exists(project_root):
+        print_error(f"Evolution state has been saved. {remedy}")
+        return
+    print_error(
+        "No evolution state was saved: the failure happened before the first "
+        "generation completed, so there is nothing to resume. "
+        + remedy.replace("[cyan]helix resume[/cyan]", "[cyan]helix evolve[/cyan]")
+    )
+
+
 def _print_cleanup_hint() -> None:
     print_info(
         "HELIX worktrees and saved state remain on disk after the run. "
@@ -695,6 +722,16 @@ def evolve(
             "Run [cyan]helix resume[/cyan] to continue when rate limits clear."
         )
         raise SystemExit(2)
+    except CredentialRefreshError as exc:
+        # Every in-loop path handles this itself (the slot is skipped and the
+        # run continues), so reaching here means a path that does not -- in
+        # practice seedless seed generation, which runs before any state has
+        # been saved.  Show the panel with its suggestion instead of a raw
+        # traceback, and only promise a resume when there is a state file.
+        logger.error("Credential failure escaped the evolution loop: %s", exc)
+        print_helix_error(exc)
+        _print_credential_failure_hint(project_root, config.agent.backend, exc)
+        raise SystemExit(2)
     except KeyboardInterrupt:
         _handle_keyboard_interrupt(project_root)
     else:
@@ -1237,6 +1274,11 @@ def resume(config_path: str, project_dir: Path | None) -> None:
             "Run [cyan]helix resume[/cyan] again when rate limits clear."
         )
         raise SystemExit(2)
+    except CredentialRefreshError as exc:
+        logger.error("Credential failure escaped the resumed loop: %s", exc)
+        print_helix_error(exc)
+        _print_credential_failure_hint(project_root, config.agent.backend, exc)
+        raise SystemExit(2)
     except KeyboardInterrupt:
         _handle_keyboard_interrupt(project_root)
     else: