diff --git a/CLAUDE.md b/CLAUDE.md index 1a9ae04ec..9ed77fb63 100644 --- a/CLAUDE.md +++ b/CLAUDE.md @@ -209,60 +209,11 @@ Usage: `factory ceo /path --profile litellm-proxy` ## Runners -The factory supports multiple CLI backends via the runner abstraction (`factory/runners/`). By default, it uses Claude Code (`claude` CLI). Bob Shell (`bob` CLI) and OpenAI Codex (`codex` CLI) are also supported as switchable alternatives. +The factory uses Claude Code (`claude` CLI) as its runner backend (`factory/runners/`). Third-party runners can be registered via Python entry points (`factory.runners` group). -**Runner selection:** Set `FACTORY_RUNNER=codex` (or `bob`) to switch backends, or pass `--runner codex` to individual commands. Default is `claude`. +**Runner selection:** Default is `claude`. Set `FACTORY_RUNNER` env var or pass `--runner` to individual commands if using a plugin runner. -**Bob Shell specifics:** -- Requires `BOBSHELL_API_KEY` environment variable to be set -- Uses 'code' mode; agent role definitions are injected via the prompt -- Model selection is not configurable (Bob Shell uses its default model) - -**Dry-run mode:** Set `FACTORY_BOB_DRY_RUN=1` to test Bob Shell integration without spending tokens. The factory returns stub responses and logs usage. This is automatically set in tests via `tests/conftest.py`. - -**Token guardrails:** Bob Shell has no token telemetry, so the factory self-enforces invocation ceilings: -- `FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE` (default: 8) -- All invocations are logged to `.factory/bob_usage.jsonl` -- When ≤2 invocations remain before the ceiling, a warning is logged and emitted to `.factory/events.jsonl` (type: `bob.ceiling_warning`) -- Ceiling violations emit events to `.factory/events.jsonl` and abort with an actionable error message - -**Codex specifics:** -- Requires `CODEX_API_KEY` (or `OPENAI_API_KEY`) environment variable (or set via config.toml profile) -- `CODEX_API_KEY` is auto-mapped to `OPENAI_API_KEY` in subprocess env if needed -- Headless mode uses `codex exec` with `--sandbox workspace-write --ask-for-approval never` -- Model selection via `--model` flag (e.g., `gpt-5.4`, `gpt-5.2-codex`) -- Progress streams to stderr, final message to stdout (matches factory capture model) -- Install: `npm install -g @openai/codex` - -**Codex dry-run mode:** Set `FACTORY_CODEX_DRY_RUN=1` to test Codex integration without spending tokens. - -**Codex config profile example** (`~/.factory/config.toml`): -```toml -[credentials.codex] -FACTORY_RUNNER = "codex" -CODEX_API_KEY = "..." -``` -Then run: `factory ceo /path/to/project --profile codex` - -**OpenCode specifics:** -- The factory targets `anomalyco/opencode` v1.x (TypeScript/Bun). Install via: `curl -fsSL https://opencode.ai/install | bash` or `npm i -g opencode-ai` -- Auth: run `opencode auth login` (interactive), or set a provider env var (`ANTHROPIC_API_KEY`, `OPENAI_API_KEY`, `AWS_ACCESS_KEY_ID`, etc.) -- Headless mode uses `opencode run '' --format json --dir --auto` -- Model selection via `--model` flag (e.g., `anthropic/claude-sonnet-4-20250514`) -- Session management: `--title ` (name a session), `--session ` (resume by ID), `--continue` (continue last session) -- Dry-run mode: `FACTORY_OPENCODE_DRY_RUN=1` -- Token guardrails: `FACTORY_OPENCODE_MAX_INVOCATIONS_PER_CYCLE` (default: 8), logged to `.factory/opencode_usage.jsonl` -- Unsupported: `--bg` (no background mode), `--tmux-persist` (returns explicit error), CEO message events (no JSON streaming equivalent) - -**OpenCode config profile example** (`~/.factory/config.toml`): -```toml -[credentials.opencode] -FACTORY_RUNNER = "opencode" -ANTHROPIC_API_KEY = "sk-ant-..." -``` -Then run: `factory ceo /path/to/project --profile opencode` - -**Important:** Target projects should add `.factory/` to their `.gitignore`. The factory writes experiment data, usage logs, and potentially sensitive auth files (`.factory/.bob_auth`) to this directory. These are project-local artifacts that should not be committed to version control. +**Important:** Target projects should add `.factory/` to their `.gitignore`. The factory writes experiment data and usage logs to this directory. These are project-local artifacts that should not be committed to version control. ## Running the factory @@ -347,7 +298,7 @@ factory precheck /path --score-before 0.7 --score-after 0.85 # Hard precheck ga factory review --verdict KEEP --pr 42 # Post structured review on GitHub PR ``` -`factory run` / `factory ceo` spawn the CEO agent as a subprocess using the selected runner (`claude` by default, or `bob` with `--runner bob`). The CEO owns the full workflow: state detection, agent spawning, experiment lifecycle, and mandatory archival. The `--loop` flag adds a heartbeat wrapper with configurable interval and max cycles. `--mode meta` runs the full Improve loop on the factory itself, then ACE playbook evolution for all agent roles. `--focus` activates targeted mode: builds exactly one item and exits. Accepts backlog names (`--focus "eval reliability"`), issue numbers (`--focus 42`), issue URLs, or `owner/repo#N` shorthand. Multiple issues can be specified in a single `--focus` string using commas, spaces, or "and" (e.g., `--focus "111 and 112"`, `--focus "issue 42, issue 43"`, `--focus "#111 #112"`). Each issue is fetched independently and added as a separate backlog item. Issue refs are auto-detected and fetched via `gh`/`glab` CLI. Works in improve, research, and create modes; mutually exclusive with `--loop`. In create mode, `--focus` provides the mode description; use `--focus "mode_name: change description"` to update an existing registered mode instead of creating a new one. `--mode design` enters ideation mode. For new ideas (e.g. `factory ceo "distributed eval runner" --mode design`), the CEO researches the space via the Researcher, then iteratively refines the idea with the Strategist through user feedback, producing a phased build plan before building. For existing projects (e.g. `factory ceo /path/to/project --mode design`), the CEO studies the project (backlog, eval scores, open issues, history), presents findings, and discusses what to work on, then continues to implementation automatically after approval. `--mode interactive` is accepted as a backward-compatible alias for `--mode design`. `--focus` is allowed on existing projects to seed the discussion topic. Incompatible with `--headless` unless `--auto-approve` is used. `--auto-approve` lifts the headless restriction for design mode, forcing headless execution and auto-approving user gates (e.g. strategy review) — useful for CI/CD and automated pipelines. `--from-plan ` loads an existing plan into design mode, skipping the research phase. Accepts a local file path, GitHub issue URL, issue number, or fuzzy search string (searches GitHub issues with the `plan` label). Requires `--mode design`; mutually exclusive with `--focus` and `--prompt`. When fetching from a GitHub issue, includes both the issue body and all comments. `--mode research` enters research ideation for new projects (e.g. `factory ceo "SWE-bench solver" --mode research`) — the Strategist collects research config (target metric, mutable/fixed surfaces, constraints) before building. For existing projects with `research_target` configured, runs the research improvement loop directly. Incompatible with `--headless` (for new projects) and `--prompt`. `--refine ""` enters refinement mode — routes a single change request through the Refiner → Builder → full review pipeline. Mutually exclusive with `--mode`, `--prompt`, and `--focus`. Requires an existing project directory. In foreground mode, the CEO also enters the refinement loop automatically after completing a build/improve cycle, staying active for follow-up requests without `--refine`. `--mode founder` enters rapid prototyping mode — a stripped-down pipeline (Study → Strategist → Builder → health gate → record) with 2 agent calls and 1 test run. Skips research, code review, adversarial QA, and eval scoring. Designed for fast hypothesis iteration: test an idea, see if it works, pivot. Terminal mode — does not chain to other modes. Not for production use; run `--mode improve` afterward to harden what works. Compatible with `--focus` and `--loop`. `--just-plan` (requires `--mode design`) enters planning-only mode — research + strategy + optional GitHub publishing with no implementation. Three parallel researchers investigate domain, practices, and constraints. The Strategist synthesizes a phased plan. Single user gate: keep the plan? Approval auto-publishes to GitHub as an issue with the `plan` label and seeds the backlog with plan phases. Terminal mode — does not chain to other modes. Compatible with `--focus`. Mutually exclusive with `--from-plan` and `--prompt`. +`factory run` / `factory ceo` spawn the CEO agent as a subprocess using the Claude Code runner. The CEO owns the full workflow: state detection, agent spawning, experiment lifecycle, and mandatory archival. The `--loop` flag adds a heartbeat wrapper with configurable interval and max cycles. `--mode meta` runs the full Improve loop on the factory itself, then ACE playbook evolution for all agent roles. `--focus` activates targeted mode: builds exactly one item and exits. Accepts backlog names (`--focus "eval reliability"`), issue numbers (`--focus 42`), issue URLs, or `owner/repo#N` shorthand. Multiple issues can be specified in a single `--focus` string using commas, spaces, or "and" (e.g., `--focus "111 and 112"`, `--focus "issue 42, issue 43"`, `--focus "#111 #112"`). Each issue is fetched independently and added as a separate backlog item. Issue refs are auto-detected and fetched via `gh`/`glab` CLI. Works in improve, research, and create modes; mutually exclusive with `--loop`. In create mode, `--focus` provides the mode description; use `--focus "mode_name: change description"` to update an existing registered mode instead of creating a new one. `--mode design` enters ideation mode. For new ideas (e.g. `factory ceo "distributed eval runner" --mode design`), the CEO researches the space via the Researcher, then iteratively refines the idea with the Strategist through user feedback, producing a phased build plan before building. For existing projects (e.g. `factory ceo /path/to/project --mode design`), the CEO studies the project (backlog, eval scores, open issues, history), presents findings, and discusses what to work on, then continues to implementation automatically after approval. `--mode interactive` is accepted as a backward-compatible alias for `--mode design`. `--focus` is allowed on existing projects to seed the discussion topic. Incompatible with `--headless` unless `--auto-approve` is used. `--auto-approve` lifts the headless restriction for design mode, forcing headless execution and auto-approving user gates (e.g. strategy review) — useful for CI/CD and automated pipelines. `--from-plan ` loads an existing plan into design mode, skipping the research phase. Accepts a local file path, GitHub issue URL, issue number, or fuzzy search string (searches GitHub issues with the `plan` label). Requires `--mode design`; mutually exclusive with `--focus` and `--prompt`. When fetching from a GitHub issue, includes both the issue body and all comments. `--mode research` enters research ideation for new projects (e.g. `factory ceo "SWE-bench solver" --mode research`) — the Strategist collects research config (target metric, mutable/fixed surfaces, constraints) before building. For existing projects with `research_target` configured, runs the research improvement loop directly. Incompatible with `--headless` (for new projects) and `--prompt`. `--refine ""` enters refinement mode — routes a single change request through the Refiner → Builder → full review pipeline. Mutually exclusive with `--mode`, `--prompt`, and `--focus`. Requires an existing project directory. In foreground mode, the CEO also enters the refinement loop automatically after completing a build/improve cycle, staying active for follow-up requests without `--refine`. `--mode founder` enters rapid prototyping mode — a stripped-down pipeline (Study → Strategist → Builder → health gate → record) with 2 agent calls and 1 test run. Skips research, code review, adversarial QA, and eval scoring. Designed for fast hypothesis iteration: test an idea, see if it works, pivot. Terminal mode — does not chain to other modes. Not for production use; run `--mode improve` afterward to harden what works. Compatible with `--focus` and `--loop`. `--just-plan` (requires `--mode design`) enters planning-only mode — research + strategy + optional GitHub publishing with no implementation. Three parallel researchers investigate domain, practices, and constraints. The Strategist synthesizes a phased plan. Single user gate: keep the plan? Approval auto-publishes to GitHub as an issue with the `plan` label and seeds the backlog with plan phases. Terminal mode — does not chain to other modes. Compatible with `--focus`. Mutually exclusive with `--from-plan` and `--prompt`. ## Contained runtimes diff --git a/SPEC.md b/SPEC.md index de06f410c..4c20f604b 100644 --- a/SPEC.md +++ b/SPEC.md @@ -22,7 +22,7 @@ It solves these operational problems: - It **tracks experiment outcomes as append-only history** instead of losing context on what was tried, why it was reverted, and what was learned - It **enforces eval-driven decisions** through weighted composite scores across hygiene, growth, and project-specific dimensions instead of subjective "looks good" judgments - It **preserves cross-project knowledge** in a structured archive instead of requiring each project to re-learn the same patterns and anti-patterns -- It **supports pluggable CLI backends** (Claude Code, Bob Shell, OpenAI Codex, OpenCode) through a runner abstraction instead of hard-coding a single LLM provider +- It **supports pluggable CLI backends** (Claude Code) through a runner abstraction instead of hard-coding a single LLM provider **Important boundary:** re:factory is NOT responsible for training models, hosting infrastructure, or managing authentication to LLM providers. It delegates to authenticated CLI tools and expects the user to configure credentials externally. @@ -38,7 +38,7 @@ It solves these operational problems: - Apply FEEC priority heuristic (Fix > Exploit > Explore > Combine) to classify hypotheses and detect stuck patterns after 3+ consecutive same-category reverts - Maintain cross-project knowledge archives (patterns, decisions, experiments) for domain transfer - Evolve agent playbooks automatically via ACE (Autonomous Capability Evolution) based on performance reports -- Support multiple CLI backends (Claude Code, Bob Shell, OpenAI Codex, OpenCode) through a runner abstraction +- Support multiple CLI backends (Claude Code) through a runner abstraction - Enable research mode with inner/outer loop plateau detection, adversarial GAN-style eval loops, and mutable/fixed surface constraints ### 2.2 Non-Goals @@ -80,10 +80,7 @@ re:factory treats software improvement as a scientific experiment loop: observe, ### 4.2 External Dependencies -- `claude` CLI — Claude Code runner (default) (REQUIRED unless using alternate runner) -- `bob` CLI — Bob Shell runner (OPTIONAL) -- `codex` CLI — OpenAI Codex runner (OPTIONAL) -- `opencode` CLI — OpenCode runner (OPTIONAL, requires `opencode-ai/opencode` v0.x from GitHub) +- `claude` CLI — Claude Code runner (REQUIRED) - `gh` CLI — GitHub issue fetching for `--focus` mode (OPTIONAL) - `glab` CLI — GitLab issue fetching for `--focus` mode (OPTIONAL) - `uv` — Python package manager and virtual environment tool (REQUIRED) @@ -643,8 +640,8 @@ QA agent MUST write a verdict file to `.factory/reviews/ceo-verdict-qa.md` in th re:factory uses a five-tier configuration precedence chain (highest to lowest priority): -1. **CLI flag** — e.g., `--runner codex`, `--model gpt-5.4` -2. **Environment variable** — e.g., `FACTORY_RUNNER=codex`, `ANTHROPIC_API_KEY=...` +1. **CLI flag** — e.g., `--model claude-opus-4-7` +2. **Environment variable** — e.g., `FACTORY_RUNNER=claude`, `ANTHROPIC_API_KEY=...` 3. **Profile credential** — from `~/.factory/config.toml` `[credentials.]` section (loaded via `--profile `) 4. **Config.toml default** — from `~/.factory/config.toml` `[defaults]` section 5. **Hardcoded default** — built into the code (e.g., `runner="claude"`, `model=None`) @@ -657,7 +654,7 @@ Credential profiles inject all keys from `[credentials.]` into the subp ```toml [defaults] -runner = "claude" # Default runner: "claude", "bob", "codex", "opencode" +runner = "claude" # Default runner: "claude" model = "" # Default model (empty = runner's default) projects_dir = "~/factory-projects" # Default project storage @@ -665,9 +662,6 @@ projects_dir = "~/factory-projects" # Default project storage FACTORY_RUNNER = "claude" ANTHROPIC_API_KEY = "sk-ant-..." -[credentials.codex] -FACTORY_RUNNER = "codex" -CODEX_API_KEY = "..." ``` #### Project Config (`.factory/config.json`) @@ -684,7 +678,7 @@ See Section 6.2 for full `FactoryConfig` schema. Key fields: ### 10.3 Validation and Error Surface **User config validation:** -- `runner` MUST be one of: "claude", "bob", "codex", "opencode" +- `runner` MUST be "claude" (or a third-party runner registered via entry points) - `projects_dir` MUST expand to a valid absolute path (tilde expansion allowed) - Profile sections MUST have unique names - Credential keys MUST be valid environment variable names (uppercase, underscores) @@ -817,7 +811,7 @@ See Section 6.2 for full `FactoryConfig` schema. Key fields: ### 13.3 Secret Handling **Secret sources:** -- Environment variables (`ANTHROPIC_API_KEY`, `CODEX_API_KEY`, `FACTORY_RUNNER`, etc.) +- Environment variables (`ANTHROPIC_API_KEY`, `FACTORY_RUNNER`, etc.) - Credential profiles in `~/.factory/config.toml` - `.env` files in the project directory (if present) @@ -922,7 +916,7 @@ A conforming re:factory implementation MUST satisfy these criteria: **Location:** `factory/runners/` directory -**Mechanism:** Each runner is a Python module (e.g., [[graph:factory/runners/claude.py]], [[graph:factory/runners/bob.py]], [[graph:factory/runners/codex.py]]) that implements the `spawn()` function: +**Mechanism:** Each runner is a Python module (e.g., [[graph:factory/runners/claude.py]]) that implements the `spawn()` function: ```python def spawn( diff --git a/docs/codex-mcp.md b/docs/codex-mcp.md deleted file mode 100644 index 5c48bc6fd..000000000 --- a/docs/codex-mcp.md +++ /dev/null @@ -1,78 +0,0 @@ -# Codex CLI: MCP Server Setup - -The Factory exposes its tools via the Model Context Protocol (MCP). This allows Codex CLI to call factory commands directly as tool invocations. - -## Quick Start - -```bash -codex mcp add factory -- factory serve-mcp -``` - -This registers the factory MCP server with Codex CLI. The server runs over stdio and exposes all factory subcommands as MCP tools. - -## Manual Configuration - -Add to `~/.codex/config.toml`: - -```toml -[mcp_servers.factory] -command = "factory" -args = ["serve-mcp"] -``` - -## Prerequisites - -The `factory` CLI must be installed and on PATH: - -```bash -uv tool install remote-factory -# or from source -uv tool install git+https://github.com/akashgit/remote-factory -``` - -Verify with: - -```bash -factory --help -factory serve-mcp # should start and wait for MCP messages on stdin -``` - -## Available Tools - -The MCP server exposes these factory operations: - -| Tool | Description | -|------|-------------| -| `detect` | Detect project state | -| `discover` | Introspect project and generate eval profile | -| `eval` | Run project evaluations | -| `begin` | Start a new experiment | -| `finalize` | Finalize an experiment with a verdict | -| `history` | Show experiment history | -| `status` | Print project status summary | -| `study` | Analyze codebase and write observations | -| `backlog-list` | List pending backlog items | -| `backlog-add` | Add a backlog item | -| `backlog-remove` | Remove a backlog item | - -## Installing Codex Agents - -To install factory specialist agents for direct invocation: - -```bash -factory install --runner codex -``` - -This writes TOML agent files to `~/.codex/agents/factory-*.toml`. Use them with: - -```bash -codex --agent factory-researcher -codex --agent factory-builder -codex --agent factory-ceo -``` - -## Troubleshooting - -**MCP server not found:** Ensure `factory` is on your PATH. Run `which factory` to verify. - -**Connection refused:** The MCP server uses stdio transport, not HTTP. It reads from stdin and writes to stdout. Codex CLI handles the connection automatically when configured via `codex mcp add` or `config.toml`. diff --git a/docs/runner-v2-spec.md b/docs/runner-v2-spec.md index 3c876fca9..f72936a7e 100644 --- a/docs/runner-v2-spec.md +++ b/docs/runner-v2-spec.md @@ -14,10 +14,7 @@ CLI / invoke_agent() ├── headless(request) → AgentRunResult └── interactive_run(request) → int │ - ├── ClaudeRunner ── system prompt via --append-system-prompt-file - ├── BobRunner ── concatenated prompt - ├── CodexRunner ── concatenated prompt, CODEX_HOME isolation - └── OpenCodeRunner ── concatenated prompt, PATH auto-discovery + └── ClaudeRunner ── system prompt via --append-system-prompt-file │ ▼ run_subprocess() (shared executor) @@ -51,7 +48,7 @@ CLI / invoke_agent() |-------|------|---------|-------------| | stdout | str | — | Captured output | | return_code | int | — | Process exit code | -| usage | AgentUsage \| None | None | Token telemetry (Claude only) | +| usage | AgentUsage \| None | None | Token telemetry | | metadata | dict[str, object] | {} | stderr, runner-specific data | ### RunnerMeta (factory/runners/protocol.py) @@ -83,107 +80,37 @@ myrunner = "my_package:MyRunner" Discovery: `importlib.metadata.entry_points(group="factory.runners")` -- Built-in runners (`claude`, `bob`, `codex`, `opencode`) registered in `_RUNNERS` dict +- Built-in runner (`claude`) registered in `_RUNNERS` dict - Entry points loaded once via `_load_entrypoints()` with `_entrypoints_loaded` guard - Built-in runners take precedence on name collision - Load failures logged at debug level, do not crash - CLI choices generated dynamically from `get_available_runners().keys()` -## Capability Matrix - -### E2E Tested (22 tests, all PASS, real API calls) - -| Test | Claude | Bob | Codex | OpenCode | -|------|--------|-----|-------|----------| -| Agent invocation (invoke_agent) | PASS | PASS | PASS | PASS | -| Builder makes code changes | PASS | PASS | PASS | PASS | -| Output captured to .factory/reviews/ | PASS | PASS | PASS | PASS | -| Cross-runner parity | PASS | PASS | PASS | PASS | -| Timeout handling | PASS | PASS | PASS | PASS | -| factory agent --runner CLI | PASS | PASS | PASS | PASS | -| Headless produces output | PASS | PASS | PASS | PASS | -| tmux_persist degradation | — | PASS | PASS | PASS | -| Token telemetry | PASS | — | — | — | -| factory eval | PASS | PASS | PASS | PASS | - -### Feature Matrix - -| Feature | Claude | Codex | Bob | OpenCode | -|---------|--------|-------|-----|----------| -| Headless mode | -p task | codex exec prompt | -p prompt | -p prompt -q | -| System prompt (proper slot) | --append-system-prompt-file | AGENTS.md (project-level only) | Concatenated | Concatenated | -| Model override | --model | --model (API key mode) | Not supported | --model | -| Permissions skip | --dangerously-skip-permissions | --sandbox workspace-write | --yolo | --dangerously-skip-permissions | -| Token telemetry | JSON usage block | None | None | None | -| JSON output | --output-format json | None | None | None | -| Session naming | --name | None | None | None | -| tmux persistence | Yes | Warns + fallback | Warns + fallback | Warns + fallback | -| Invocation ceilings | None | None | usage.py | None | - -### Auth Matrix - -| Runner | Primary Auth | Fallback | Config Location | -|--------|-------------|----------|-----------------| -| Claude | Vertex AI / API key / OAuth | claude CLI handles it | ~/.claude/ | -| Codex | ChatGPT OAuth (~/.codex/auth.json) | OPENAI_API_KEY (needs tool-use scopes) | ~/.codex/ | -| Bob | ~/.bob/settings.json | BOBSHELL_API_KEY env → .factory/.bob_auth file | ~/.bob/ | -| OpenCode | OPENAI_API_KEY env | Shell sourcing from ~/.zshrc | opencode config | +## ClaudeRunner Feature Matrix -## System Prompt Handling +| Feature | Claude | +|---------|--------| +| Headless mode | -p task | +| System prompt (proper slot) | --append-system-prompt-file | +| Model override | --model | +| Permissions skip | --dangerously-skip-permissions | +| Token telemetry | JSON usage block | +| JSON output | --output-format json | +| Session naming | --name | +| tmux persistence | Yes | +| Background dispatch | Yes (--bg) | -### Current Behavior +## System Prompt Handling The factory agent system has two levels of prompts: -1. **Project-level instructions** (CLAUDE.md, AGENTS.md) — read automatically by each CLI from the project directory. All runners handle this natively. - -2. **Per-agent role prompts** (e.g., "You are the Researcher agent...") — resolved by `factory/agents/runner.py` via `resolve_prompt(role, project_path)`. This is where runners diverge: - -| Runner | How agent prompt is delivered | Quality impact | -|--------|------------------------------|----------------| -| Claude | `--append-system-prompt-file` → system prompt slot | Full — model treats it as system instructions | -| Codex | Concatenated: `"{prompt}\n\n---\n\n## Current Task\n\n{task}"` | Degraded — model sees it as user message | -| Bob | Same concatenation | Degraded | -| OpenCode | Same concatenation | Degraded | - -### Mitigation - -The concatenation approach works — all 22 e2e tests pass, and agents produce useful output with all runners. The clear separator (`---\n\n## Current Task`) helps models distinguish the system instructions from the task. But Claude will have an edge on complex multi-step agent tasks where system prompt prioritization matters. - -### Future Improvements - -- **Codex**: Could write agent prompt to a temporary AGENTS.md in the project directory before invocation. Codex reads AGENTS.md automatically and treats it as system-level instructions. -- **OpenCode**: Could create a temporary opencode agent config with the system prompt. OpenCode supports custom agents with configurable system prompts. -- **Bob**: No known mechanism for separate system prompts. Concatenation is the only option. - -## Known Limitations +1. **Project-level instructions** (CLAUDE.md) — read automatically by Claude Code from the project directory. -1. **Codex OAuth + OPENAI_API_KEY conflict**: If `OPENAI_API_KEY` is in the env (e.g., set for OpenCode), Codex switches to API key mode. If that key lacks tool-use scopes → 401. The factory handles this by only setting `CODEX_HOME` in API key mode, and the test suite strips `OPENAI_API_KEY` for Codex CLI tests. - -2. **Codex model selection**: OAuth mode uses Codex default model (gpt-5.5); model override only works in API key mode. - -3. **OpenCode binary PATH**: Installed via `go install` to `~/go/bin/opencode`. Not on system PATH by default. The runner auto-detects common locations. - -4. **OpenCode Go vs npm incompatibility**: The OpenCode runner requires the **Go binary** (`go install github.com/opencode-ai/opencode@latest`). The npm package (`opencode-ai`) exposes a different CLI that does not support the `-p`, `-c`, or `-q` flags used by the runner, and will fail silently. The runner performs a runtime compatibility check on first invocation by running `opencode version` and looking for Go-style semver output (e.g. `opencode version v0.0.55`). A warning is logged if the binary appears to be the npm version. - -5. **System prompt degradation**: Non-Claude runners concatenate system + task prompts. Works but less effective than proper system prompt slot. - -6. **No fallback chains**: If a runner fails, the factory aborts. No automatic failover to another runner. - -7. **Bob invocation ceilings**: Bob Shell has no token telemetry, so the factory self-enforces invocation ceilings via `FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE` (default: 8). All invocations logged to `.factory/bob_usage.jsonl`. Ceiling violations abort with an actionable error. +2. **Per-agent role prompts** (e.g., "You are the Researcher agent...") — resolved by `factory/agents/runner.py` via `resolve_prompt(role, project_path)`. Delivered via `--append-system-prompt-file` into Claude's system prompt slot. ## Dry-Run Mode -Each runner supports a dry-run env var for testing without spending tokens: - -| Runner | Env Var | Behavior | -|--------|---------|----------| -| Claude | — | No dry-run (use mocked subprocess in tests) | -| Bob | `FACTORY_BOB_DRY_RUN=1` | Returns stub response, logs usage | -| Codex | `FACTORY_CODEX_DRY_RUN=1` | Returns stub response | -| OpenCode | `FACTORY_OPENCODE_DRY_RUN=1` | Returns stub response | - -Stub responses generated by `make_dry_run_result()` in `factory/runners/_subprocess.py`. +Claude runner uses mocked subprocess in tests (no dedicated dry-run env var). ## Files @@ -195,8 +122,5 @@ Stub responses generated by `make_dry_run_result()` in `factory/runners/_subproc | factory/runners/_subprocess.py | Shared subprocess executor, make_dry_run_result | | factory/runners/_stream.py | Streaming output, ANSI stripping | | factory/runners/claude.py | ClaudeRunner | -| factory/runners/bob.py | BobRunner with auth + ceilings | -| factory/runners/codex.py | CodexRunner with CODEX_HOME auth isolation | -| factory/runners/opencode.py | OpenCodeRunner with PATH auto-discovery | -| tests/test_runner_e2e.py | 22 e2e tests with real API calls | -| tests/test_runners.py | Unit tests (mocked subprocess, ~63 tests) | +| tests/test_runner_e2e.py | E2E tests with real API calls | +| tests/test_runners.py | Unit tests (mocked subprocess) | diff --git a/factory/agents/plugin.py b/factory/agents/plugin.py index 44c615988..ab0481fa8 100644 --- a/factory/agents/plugin.py +++ b/factory/agents/plugin.py @@ -1,4 +1,4 @@ -"""Plugin agent generation — produce Claude Code and Codex CLI agent files from source prompts.""" +"""Plugin agent generation — produce Claude Code agent files from source prompts.""" from __future__ import annotations @@ -15,10 +15,6 @@ _AGENTS_YML = Path(__file__).parent / "agents.yml" _PLUGIN_AGENTS_DIR_CANDIDATE = Path(__file__).resolve().parent.parent.parent / "agents" _PLUGIN_AGENTS_DIR: Path | None = _PLUGIN_AGENTS_DIR_CANDIDATE if _PLUGIN_AGENTS_DIR_CANDIDATE.is_dir() else None -_CODEX_PLUGIN_AGENTS_DIR_CANDIDATE = Path(__file__).resolve().parent.parent.parent / "codex-agents" -_CODEX_PLUGIN_AGENTS_DIR: Path | None = ( - _CODEX_PLUGIN_AGENTS_DIR_CANDIDATE if _CODEX_PLUGIN_AGENTS_DIR_CANDIDATE.is_dir() else None -) @dataclass(frozen=True) @@ -108,84 +104,6 @@ def _sandbox_mode(role: str) -> str: return "read-only" -def _escape_toml_multiline_literal(text: str) -> str: - """Escape text for a TOML multiline literal string (triple single-quoted). - - TOML literal strings have no escape sequences and no concatenation operator, - so triple single-quotes cannot appear inside them at all. We lossy-replace - ''' with '' (virtually never appears in agent prompts). - """ - return text.replace("'''", "''") - - -def generate_codex_agent_toml(role: str) -> str: - """Generate a TOML agent file for Codex CLI. - - Reads the same agents.yml + prompts/*.md sources as generate_agent_content - but emits TOML with fields: name, description, developer_instructions, sandbox_mode. - """ - config = load_agent_config() - if role not in config: - raise ValueError(f"Unknown agent role: {role!r}") - - meta = config[role] - prompt = (_PROMPTS_DIR / f"{role}.md").read_text() - playbook_path = _PLAYBOOKS_DIR / f"{role}.md" - if playbook_path.exists(): - playbook = playbook_path.read_text().strip() - if playbook: - prompt = inject_playbook(prompt, playbook) - - sandbox = _sandbox_mode(role) - escaped_desc = ( - meta.description.replace("\\", "\\\\").replace('"', '\\"') - .replace("\n", " ").replace("\t", " ") - ) - escaped_prompt = _escape_toml_multiline_literal(prompt) - - return ( - f'# GENERATED FILE — do not edit directly.\n' - f'# Source: factory/agents/prompts/{role}.md\n' - f'# Run: python scripts/sync_agents.py\n' - f'\n' - f'name = "factory-{role}"\n' - f'description = "{escaped_desc}"\n' - f'sandbox_mode = "{sandbox}"\n' - f'\n' - f"developer_instructions = '''\n" - f'> **Prerequisite:** The `factory` CLI must be on PATH.\n' - f'> Install: `uv tool install remote-factory`\n' - f'\n' - f"{escaped_prompt}'''\n" - ) - - -def check_codex_agents_in_sync(agents_dir: Path | None = None) -> list[str]: - """Compare generated Codex TOML agent files against what's on disk. - - Returns a list of role names that are out of sync (empty = all good). - """ - if agents_dir is None: - agents_dir = _CODEX_PLUGIN_AGENTS_DIR - if agents_dir is None: - return [] - - config = load_agent_config() - out_of_sync: list[str] = [] - for role in config: - expected = generate_codex_agent_toml(role) - agent_path = agents_dir / f"{role}.toml" - - if not agent_path.exists(): - out_of_sync.append(role) - continue - - if agent_path.read_text() != expected: - out_of_sync.append(role) - - return out_of_sync - - def check_agents_in_sync(agents_dir: Path | None = None) -> list[str]: """Compare generated agent files against what's on disk. diff --git a/factory/agents/prompts/evolver.md b/factory/agents/prompts/evolver.md deleted file mode 100644 index 2ccdd80ff..000000000 --- a/factory/agents/prompts/evolver.md +++ /dev/null @@ -1,38 +0,0 @@ -# Evolver Agent - -You are the Evolver — a specialist that synthesizes new workflow designs from reflection insights and evolutionary pressure. - -## Task - -Given a parent workflow, a ReflectionReport, and the current evolutionary state, propose specific mutations that improve the workflow's benchmark performance. - -## Input - -- **Parent workflow**: The current best workflow DAG (nodes, edges, start_node) -- **ReflectionReport**: Contrastive analysis of what works vs what doesn't -- **Generation stats**: Current best score, diversity, archive coverage - -## Output - -Produce a list of specific, actionable mutations: - -```json -{ - "mutations": [ - { - "operator": "NODE_INSERT", - "target_node": "builder", - "rationale": "Reflection shows winners have a researcher before builder", - "details": {"new_role": "researcher", "insert_after": "study"} - } - ] -} -``` - -## Rules - -1. Prioritize mutations suggested by the ReflectionReport -2. Each mutation must be implementable by one of the 7 operators: NODE_INSERT, NODE_REMOVE, EDGE_REDIRECT, PARALLELIZE, SERIALIZE, PARAM_MUTATE, PROMPT_MUTATE -3. Keep workflows under 30 nodes — if the parent is already large, prefer PARAM_MUTATE or NODE_REMOVE -4. Maintain at least 20% random mutations for diversity — don't over-exploit reflection -5. Ground every rationale in specific data from the reflection or generation stats diff --git a/factory/agents/runner.py b/factory/agents/runner.py index a4dcebf21..5c315d282 100644 --- a/factory/agents/runner.py +++ b/factory/agents/runner.py @@ -32,7 +32,7 @@ def __init__(self, failure_count: int, last_agent: str) -> None: f"Aborting after {failure_count} consecutive agent spawn failures. " f"Last failed agent: {last_agent}. " "Check .factory/events.jsonl for details. " - "This usually means BOBSHELL_API_KEY is not being propagated to subprocesses." + "This usually means ANTHROPIC_API_KEY is not being propagated to subprocesses." ) diff --git a/factory/ceo_completion.py b/factory/ceo_completion.py index 94d6b2836..b17a74c01 100644 --- a/factory/ceo_completion.py +++ b/factory/ceo_completion.py @@ -413,12 +413,7 @@ def _build_continuation_task(gap: IncompleteGap, cycle_state: CycleState | None def _budget_allows_respawn(runner_name: str | None, project_path: Path) -> bool: - """Check if budget/ceiling allows another spawn. - - With only per-cycle limits (no daily/session limit), we can always start - a new cycle. The per-cycle limit is enforced within BobRunner during execution. - """ - # All runners can respawn - per-cycle limits are enforced within the cycle + """Check if budget/ceiling allows another spawn.""" return True diff --git a/factory/cli/_parser_groups.py b/factory/cli/_parser_groups.py index e0efcf2f4..1f426244d 100644 --- a/factory/cli/_parser_groups.py +++ b/factory/cli/_parser_groups.py @@ -254,18 +254,12 @@ def add_self_evolution_parsers(sub: argparse._SubParsersAction) -> None: # type def add_configuration_parsers(sub: argparse._SubParsersAction) -> None: # type: ignore[type-arg] sub.add_parser("self-update", help="Upgrade the factory CLI to the latest version") - p = sub.add_parser("install", help="Install Factory agents as CLI agents (~/.claude/agents/ or ~/.codex/agents/)") + p = sub.add_parser("install", help="Install Factory agents as Claude Code CLI agents (~/.claude/agents/)") p.add_argument( "--role", default=None, help="Install only a specific agent role (default: all)", ) - p.add_argument( - "--runner", - choices=["claude", "codex"], - default="claude", - help="Target CLI: claude writes Markdown to ~/.claude/agents/, codex writes TOML to ~/.codex/agents/ (default: claude)", - ) p = sub.add_parser("usage", help="Show per-agent token usage and cost breakdown") p.add_argument("path", help="Path to the project") diff --git a/factory/cli/admin.py b/factory/cli/admin.py index 31f8cf43f..24afa2b74 100644 --- a/factory/cli/admin.py +++ b/factory/cli/admin.py @@ -281,15 +281,12 @@ def cmd_self_update(args: argparse.Namespace) -> int: def cmd_install(args: argparse.Namespace) -> int: - """Install Factory agents as Claude Code or Codex CLI agents.""" + """Install Factory agents as Claude Code CLI agents.""" from factory.agents.plugin import ( generate_agent_content, - generate_codex_agent_toml, load_agent_config, ) - runner = getattr(args, "runner", "claude") or "claude" - role_filter = getattr(args, "role", None) config = load_agent_config() @@ -300,32 +297,19 @@ def cmd_install(args: argparse.Namespace) -> int: roles = [role_filter] if role_filter else list(config) - if runner == "codex": - agents_dir = Path.home() / ".codex" / "agents" - agents_dir.mkdir(parents=True, exist_ok=True) - for role in roles: - content = generate_codex_agent_toml(role) - agent_path = agents_dir / f"factory-{role}.toml" - agent_path.write_text(content) - print(f" Installed factory-{role} -> {agent_path}") - print() - print("Usage:") - print(" codex --agent factory- # from any project directory") - print(' codex --agent factory-ceo "improve X" # with initial prompt') - else: - agents_dir = Path.home() / ".claude" / "agents" - agents_dir.mkdir(parents=True, exist_ok=True) - for role in roles: - content = generate_agent_content(role) - agent_path = agents_dir / f"factory-{role}.md" - agent_path.write_text(content) - print(f" Installed factory-{role} -> {agent_path}") - print() - print("Usage:") - print(" claude --agent factory- # from any project directory") - print(' claude --agent factory-ceo "improve X" # with initial prompt') - print() - print('Or from within Claude Code, ask: "use the factory- agent"') + agents_dir = Path.home() / ".claude" / "agents" + agents_dir.mkdir(parents=True, exist_ok=True) + for role in roles: + content = generate_agent_content(role) + agent_path = agents_dir / f"factory-{role}.md" + agent_path.write_text(content) + print(f" Installed factory-{role} -> {agent_path}") + print() + print("Usage:") + print(" claude --agent factory- # from any project directory") + print(' claude --agent factory-ceo "improve X" # with initial prompt') + print() + print('Or from within Claude Code, ask: "use the factory- agent"') return 0 diff --git a/factory/runners/__init__.py b/factory/runners/__init__.py index 55f390146..4747387cd 100644 --- a/factory/runners/__init__.py +++ b/factory/runners/__init__.py @@ -1,14 +1,11 @@ -"""Runner abstraction layer for CLI backends (claude, bob, etc.).""" +"""Runner abstraction layer for CLI backends.""" from __future__ import annotations from pathlib import Path from factory.runners._stream import should_stream, stream_subprocess -from factory.runners.bob import BobRunner, is_dry_run from factory.runners.claude import ClaudeRunner -from factory.runners.codex import CodexRunner, is_codex_dry_run -from factory.runners.opencode import OpenCodeRunner, is_opencode_dry_run from factory.runners.protocol import Runner, RunnerMeta import structlog @@ -19,24 +16,15 @@ "Runner", "RunnerMeta", "ClaudeRunner", - "BobRunner", - "CodexRunner", - "OpenCodeRunner", "get_runner", "get_available_runners", "get_runner_choices", - "is_dry_run", - "is_codex_dry_run", - "is_opencode_dry_run", "should_stream", "stream_subprocess", ] _RUNNERS: dict[str, type[Runner]] = { "claude": ClaudeRunner, # type: ignore[dict-item] - "bob": BobRunner, # type: ignore[dict-item] - "codex": CodexRunner, # type: ignore[dict-item] - "opencode": OpenCodeRunner, # type: ignore[dict-item] } @@ -67,10 +55,6 @@ def get_runner(name: str | None = None, project_path: Path | None = None) -> Run available = ", ".join(_RUNNERS.keys()) raise ValueError(f"Unknown runner '{resolved}'. Available: {available}") - if resolved == "bob": - return BobRunner(project_path=project_path) - if resolved == "opencode": - return OpenCodeRunner(project_path=project_path) return _RUNNERS[resolved]() diff --git a/factory/runners/bob.py b/factory/runners/bob.py deleted file mode 100644 index 33bf414a8..000000000 --- a/factory/runners/bob.py +++ /dev/null @@ -1,316 +0,0 @@ -"""BobRunner — Bob Shell CLI backend implementation.""" - -from __future__ import annotations - -import os -import shutil -import subprocess as _subprocess -import time -from datetime import datetime, timezone -from pathlib import Path -from typing import TYPE_CHECKING - -import structlog - -from factory.runners._subprocess import run_subprocess -from factory.runners.usage import ( - CeilingExceededError, - check_ceilings, - log_usage, -) - -if TYPE_CHECKING: - from factory.models import AgentRunRequest, AgentRunResult - from factory.runners.protocol import RunnerMeta - -log = structlog.get_logger() - -_auth_checked = False - -_AUTH_FILE_NAME = ".bob_auth" - - -class BobAuthError(Exception): - """Raised when BOBSHELL_API_KEY is not set.""" - - def __init__(self) -> None: - super().__init__( - "BOBSHELL_API_KEY environment variable is not set. " - "See bob-runner-package/bob-shell-docs/README.md for setup instructions." - ) - - -def _find_auth_file(start_path: Path) -> Path | None: - """Search for the auth file starting from start_path and walking up.""" - path = start_path.resolve() - while path != path.parent: - auth_file = path / ".factory" / _AUTH_FILE_NAME - if auth_file.is_file(): - return auth_file - path = path.parent - return None - - -def _persist_key(project_path: Path) -> None: - """Persist BOBSHELL_API_KEY to a file for nested subagent spawns.""" - key = os.environ.get("BOBSHELL_API_KEY") - if not key: - return - - factory_dir = project_path / ".factory" - if not factory_dir.is_dir(): - return - - auth_file = factory_dir / _AUTH_FILE_NAME - try: - auth_file.write_text(key) - auth_file.chmod(0o600) - log.debug("bob_key_persisted", path=str(auth_file)) - except OSError as e: - log.warning("bob_key_persist_failed", error=str(e)) - - -def _check_auth(start_path: Path | None = None) -> None: - """Check that BOBSHELL_API_KEY is set (once per process).""" - global _auth_checked - if _auth_checked: - return - - if os.environ.get("BOBSHELL_API_KEY"): - _auth_checked = True - return - - search_from = start_path if start_path is not None else Path.cwd() - auth_file = _find_auth_file(search_from) - if auth_file: - try: - key = auth_file.read_text().strip() - if key: - os.environ["BOBSHELL_API_KEY"] = key - log.info("bob_key_loaded", path=str(auth_file)) - _auth_checked = True - return - except OSError as e: - log.warning("bob_auth_file_read_failed", path=str(auth_file), error=str(e)) - - bob_config = Path.home() / ".bob" / "settings.json" - if bob_config.is_file(): - log.info("bob_native_auth_detected", path=str(bob_config)) - _auth_checked = True - return - - raise BobAuthError() - - -def _has_bob_auth() -> bool: - """Check if Bob auth is available via any supported method.""" - if os.environ.get("BOBSHELL_API_KEY"): - return True - auth_file = _find_auth_file(Path.cwd()) - if auth_file is not None: - try: - if auth_file.read_text().strip(): - return True - except OSError: - pass - bob_config = Path.home() / ".bob" / "settings.json" - return bob_config.is_file() - - -def is_dry_run() -> bool: - """Return True if dry-run mode is enabled.""" - from factory.user_config import resolve - - val = resolve("bob_dry_run", env_var="FACTORY_BOB_DRY_RUN") or "" - return val.lower() in ("1", "true", "yes") - - -def _get_bob_bin_dir() -> str | None: - """Find the directory containing the bob binary.""" - bob_path = shutil.which("bob") - if bob_path: - return str(Path(bob_path).parent) - return None - - -def _make_env_with_bob_path() -> dict[str, str]: - """Create environment dict with bob's bin directory prepended to PATH.""" - env = dict(os.environ) - bob_bin_dir = _get_bob_bin_dir() - if bob_bin_dir: - current_path = env.get("PATH", "") - if not current_path.startswith(bob_bin_dir): - env["PATH"] = f"{bob_bin_dir}:{current_path}" - log.debug("bob_path_prepended", dir=bob_bin_dir) - return env - - -_BOB_CHAT_MODE = "code" - - -class BobRunner: - """Runner implementation for Bob Shell CLI.""" - - name: str = "bob" - - @classmethod - def metadata(cls) -> RunnerMeta: - from factory.runners.protocol import RunnerMeta - return RunnerMeta( - name="bob", - display_name="Bob Shell", - binary="bob", - install_hint="npm install -g bob-shell", - required_env_vars=["BOBSHELL_API_KEY"], - supports_model_override=False, - supports_usage_telemetry=False, - supports_session_name=False, - custom_auth_check=_has_bob_auth, - ) - - def __init__( - self, - cycle_start: datetime | None = None, - project_path: Path | None = None, - ) -> None: - if cycle_start is not None: - self.cycle_start = cycle_start - elif project_path is not None: - from factory.ceo_completion import read_cycle_state - - state = read_cycle_state(project_path) - self.cycle_start = state.started_at if state else datetime.now(timezone.utc) - else: - self.cycle_start = datetime.now(timezone.utc) - self._role: str = "unknown" - - def build_command(self, request: AgentRunRequest) -> tuple[list[str], dict[str, str], list[Path]]: - """Build the Bob Shell CLI command and env dict.""" - chat_mode = _BOB_CHAT_MODE - full_task = f"{request.prompt}\n\n---\n\n## Current Task\n\n{request.task}" - - cmd = ["bob", "-p", full_task, f"--chat-mode={chat_mode}"] - if request.skip_permissions: - cmd.append("--yolo") - - env = _make_env_with_bob_path() - return cmd, env, [] - - async def headless(self, request: AgentRunRequest) -> AgentRunResult: - """Run a headless Bob Shell invocation.""" - from factory.models import AgentRunResult - - tmux_persist = request.extras.get("tmux_persist", False) - if tmux_persist: - return AgentRunResult( - stdout="Error: --tmux-persist is not supported with the bob runner. Use --runner claude.", - return_code=1, - ) - background = request.extras.get("background", False) - if background: - log.warning("bob_bg_not_supported", hint="--bg is a claude-only feature") - self._role = request.role - project_path = request.project_path or self._find_project_path(request.cwd) - - _persist_key(project_path) - - if is_dry_run(): - from factory.runners._subprocess import make_dry_run_result - result = make_dry_run_result("bob", request.role, request.cwd, request.task) - log_usage(project_path, request.role, request.cwd, 0.0, 0, dry_run=True) - return result - - _check_auth(request.cwd) - - try: - check_ceilings(project_path, self.cycle_start) - except CeilingExceededError as e: - self._emit_ceiling_event(project_path, e) - return AgentRunResult(stdout=str(e), return_code=1) - - cmd, env, _ = self.build_command(request) - - log.info("bob_headless", cwd=str(request.cwd), role=request.role, chat_mode=_BOB_CHAT_MODE) - - start_time = time.monotonic() - - result = await run_subprocess( - cmd, cwd=str(request.cwd), env=env, - timeout=request.timeout, runner_name="bob", role=request.role, - sanitize=True, - ) - - duration = time.monotonic() - start_time - log_usage(project_path, request.role, request.cwd, duration, result.return_code, dry_run=False) - - return result - - def build_interactive_command(self, request: AgentRunRequest) -> tuple[list[str], dict[str, str], list[Path]]: - """Build the CLI command, env dict, and temp files for an interactive invocation.""" - chat_mode = _BOB_CHAT_MODE - full_task = f"{request.prompt}\n\n---\n\n## Current Task\n\n{request.task}" - - cmd = [ - "bob", - f"--chat-mode={chat_mode}", - "-i", full_task, - ] - if request.skip_permissions: - cmd.append("--yolo") - - env = _make_env_with_bob_path() - return cmd, env, [] - - def interactive_run(self, request: AgentRunRequest) -> int: - """Run an interactive Bob Shell session as a subprocess.""" - project_path = request.project_path or self._find_project_path(request.cwd) - - _persist_key(project_path) - - if is_dry_run(): - yolo_flag = " --yolo" if request.skip_permissions else "" - print(f"[DRY-RUN] Would run: bob --chat-mode=factory-{request.role}{yolo_flag}") - print(f"[DRY-RUN] Task: {request.task[:200]}...") - return 0 - - _check_auth(request.cwd) - - try: - check_ceilings(project_path, self.cycle_start) - except CeilingExceededError as e: - print(f"ERROR: {e}") - return 1 - - cmd, env, _ = self.build_interactive_command(request) - - log.info("bob_interactive", cwd=str(request.cwd), chat_mode=_BOB_CHAT_MODE) - - result = _subprocess.run(cmd, cwd=request.cwd, env=env) - return result.returncode - - def _find_project_path(self, cwd: Path) -> Path: - """Find the project root (directory containing .factory/).""" - path = cwd.resolve() - while path != path.parent: - if (path / ".factory").is_dir(): - return path - path = path.parent - return cwd.resolve() - - def _emit_ceiling_event(self, project_path: Path, error: CeilingExceededError) -> None: - """Emit a structured event when a ceiling is hit.""" - try: - from factory.events import emit_event - - emit_event( - project_path, - "bob.ceiling_exceeded", - data={ - "ceiling": error.ceiling_name, - "current": error.current, - "limit": error.limit, - "env_var": error.env_var, - }, - ) - except Exception: - log.debug("bob_ceiling_event_failed", exc_info=True) diff --git a/factory/runners/codex.py b/factory/runners/codex.py deleted file mode 100644 index 65b9a93d9..000000000 --- a/factory/runners/codex.py +++ /dev/null @@ -1,219 +0,0 @@ -"""CodexRunner — OpenAI Codex CLI backend implementation.""" - -from __future__ import annotations - -import asyncio -import os -import subprocess -import tempfile -from pathlib import Path -from typing import TYPE_CHECKING - -import structlog - -from factory.runners._subprocess import run_subprocess - -if TYPE_CHECKING: - from factory.models import AgentRunRequest, AgentRunResult - from factory.runners.protocol import RunnerMeta - -log = structlog.get_logger() - -_auth_checked = False - - -class CodexAuthError(Exception): - """Raised when neither CODEX_API_KEY nor OPENAI_API_KEY is set.""" - - def __init__(self) -> None: - super().__init__( - "CODEX_API_KEY (or OPENAI_API_KEY) environment variable is not set. " - "Set it directly or add it to a config.toml credential profile: " - "[credentials.codex] CODEX_API_KEY = \"...\"" - ) - - -def _has_codex_oauth() -> bool: - """Check if Codex has OAuth credentials in its default config.""" - auth_file = Path.home() / ".codex" / "auth.json" - return auth_file.is_file() - - -def _using_api_key() -> bool: - """Return True if an explicit API key is set in the environment.""" - return bool(os.environ.get("CODEX_API_KEY") or os.environ.get("OPENAI_API_KEY")) - - -def _check_auth() -> None: - """Check that Codex auth is available (OAuth preferred, then API key).""" - global _auth_checked # noqa: PLW0603 - if _auth_checked: - return - if _has_codex_oauth(): - log.info("codex_oauth_detected") - _auth_checked = True - return - if _using_api_key(): - _auth_checked = True - return - raise CodexAuthError() - - -def _make_codex_env() -> tuple[dict[str, str], tempfile.TemporaryDirectory[str] | None]: - """Build subprocess env with auth isolation. - - OAuth is preferred when ~/.codex/auth.json exists — OPENAI_API_KEY is - stripped from the env so Codex doesn't switch to API key mode (which - can cause 401 errors when the key lacks Responses API scopes). - - In API key mode, sets CODEX_HOME to a temp dir to avoid stale OAuth. - - Returns (env_dict, tmpdir_handle_or_None) — caller must keep tmpdir_handle - alive until the subprocess exits, then call .cleanup() if not None. - """ - env = {k: v for k, v in os.environ.items() if k != "VIRTUAL_ENV"} - - if _has_codex_oauth(): - env.pop("OPENAI_API_KEY", None) - env.pop("CODEX_API_KEY", None) - return env, None - - if "OPENAI_API_KEY" not in env and "CODEX_API_KEY" in env: - env["OPENAI_API_KEY"] = env["CODEX_API_KEY"] - - if _using_api_key(): - tmpdir = tempfile.TemporaryDirectory(prefix="factory-codex-") - env["CODEX_HOME"] = tmpdir.name - return env, tmpdir - - return env, None - - -def is_codex_dry_run() -> bool: - """Return True if Codex dry-run mode is enabled.""" - from factory.user_config import resolve - - val = resolve("codex_dry_run", env_var="FACTORY_CODEX_DRY_RUN") or "" - return val.lower() in ("1", "true", "yes") - - -class CodexRunner: - """Runner implementation for OpenAI Codex CLI.""" - - name: str = "codex" - - @classmethod - def metadata(cls) -> RunnerMeta: - from factory.runners.protocol import RunnerMeta - return RunnerMeta( - name="codex", - display_name="OpenAI Codex", - binary="codex", - install_hint="npm install -g @openai/codex", - required_env_vars=["OPENAI_API_KEY"], - supports_usage_telemetry=False, - supports_session_name=False, - ) - - def build_command(self, request: AgentRunRequest) -> tuple[list[str], dict[str, str], list[Path]]: - """Build the Codex CLI command, env dict, and temp files.""" - full_prompt = f"{request.prompt}\n\n---\n\n## Current Task\n\n{request.task}" - - cmd = ["codex", "exec"] - - if _using_api_key(): - cmd.append("--ignore-user-config") - - if request.skip_permissions: - cmd.extend(["--sandbox", "workspace-write"]) - - if request.model: - cmd.extend(["--model", request.model]) - - cmd.append("--skip-git-repo-check") - cmd.extend(["--", full_prompt]) - - env, tmpdir = _make_codex_env() - self._tmpdir = tmpdir - return cmd, env, [] - - async def headless(self, request: AgentRunRequest) -> AgentRunResult: - """Run a headless Codex CLI invocation via ``codex exec``.""" - from factory.models import AgentRunResult - - tmux_persist = request.extras.get("tmux_persist", False) - if tmux_persist: - return AgentRunResult( - stdout="Error: --tmux-persist is not supported with the codex runner. Use --runner claude.", - return_code=1, - ) - background = request.extras.get("background", False) - if background: - log.warning("codex_bg_not_supported", hint="--bg is a claude-only feature") - if is_codex_dry_run(): - from factory.runners._subprocess import make_dry_run_result - return make_dry_run_result("codex", request.role, request.cwd, request.task) - - _check_auth() - - cmd, env, _ = self.build_command(request) - - log.info("codex_headless", cwd=str(request.cwd), model=request.model, role=request.role) - - retried = False - try: - result = await run_subprocess( - cmd, cwd=str(request.cwd), env=env, - timeout=request.timeout, runner_name="codex", role=request.role, - ) - stderr = str(result.metadata.get("stderr", "")) - if "401 Unauthorized" in stderr and not retried: - retried = True - log.warning("codex_auth_retry", reason="401 Unauthorized in stderr") - await asyncio.sleep(2) - result = await run_subprocess( - cmd, cwd=str(request.cwd), env=env, - timeout=request.timeout, runner_name="codex", role=request.role, - ) - return result - finally: - if hasattr(self, "_tmpdir") and self._tmpdir is not None: - self._tmpdir.cleanup() - - def build_interactive_command(self, request: AgentRunRequest) -> tuple[list[str], dict[str, str], list[Path]]: - """Build the CLI command, env dict, and temp files for an interactive invocation.""" - full_prompt = f"{request.prompt}\n\n---\n\n## Current Task\n\n{request.task}" - - cmd = ["codex", full_prompt] - - if _using_api_key(): - cmd.append("--ignore-user-config") - - if request.skip_permissions: - cmd.append("--full-auto") - - if request.model: - cmd.extend(["--model", request.model]) - - env, tmpdir = _make_codex_env() - self._tmpdir = tmpdir - return cmd, env, [] - - def interactive_run(self, request: AgentRunRequest) -> int: - """Run an interactive Codex CLI session as a subprocess.""" - if is_codex_dry_run(): - print("[DRY-RUN] Would exec: codex (interactive)") - print(f"[DRY-RUN] Task: {request.task[:200]}...") - return 0 - - _check_auth() - - cmd, env, _ = self.build_interactive_command(request) - try: - log.info("codex_interactive", cwd=str(request.cwd)) - result = subprocess.run(cmd, cwd=request.cwd, env=env) - return result.returncode - finally: - if hasattr(self, "_tmpdir") and self._tmpdir is not None: - self._tmpdir.cleanup() - diff --git a/factory/runners/opencode.py b/factory/runners/opencode.py deleted file mode 100644 index 84673d3a4..000000000 --- a/factory/runners/opencode.py +++ /dev/null @@ -1,353 +0,0 @@ -"""OpenCodeRunner — OpenCode v1.x (anomalyco/opencode) CLI backend.""" - -from __future__ import annotations - -import os -import subprocess -import time -from datetime import datetime, timezone -from pathlib import Path -from typing import TYPE_CHECKING - -import structlog - -from factory.runners._subprocess import run_subprocess -from factory.runners.usage import ( - CeilingExceededError, - check_ceilings, - log_usage, -) - -if TYPE_CHECKING: - from factory.models import AgentRunRequest, AgentRunResult - from factory.runners.protocol import RunnerMeta - -log = structlog.get_logger() - -_auth_checked = False -_compat_checked = False - -_RUNNER_NAME = "opencode" - -_PROVIDER_ENV_VARS = ( - "ANTHROPIC_API_KEY", - "OPENAI_API_KEY", - "AWS_ACCESS_KEY_ID", - "GOOGLE_APPLICATION_CREDENTIALS", - "AZURE_OPENAI_API_KEY", -) - - -class OpenCodeAuthError(Exception): - """Raised when no OpenCode auth is available.""" - - def __init__(self) -> None: - super().__init__( - "No OpenCode authentication found. " - "Run 'opencode auth login' to authenticate, " - "or set a provider API key (ANTHROPIC_API_KEY, OPENAI_API_KEY, etc.). " - "Alternatively, add keys to a config.toml credential profile: " - '[credentials.opencode] ANTHROPIC_API_KEY = "..."' - ) - - -def _has_opencode_auth() -> bool: - """Check if OpenCode auth is available via config dir or provider env vars.""" - opencode_dir = Path.home() / ".opencode" - if opencode_dir.is_dir(): - return True - return any(os.environ.get(v) for v in _PROVIDER_ENV_VARS) - - -def _check_auth() -> None: - """Check that OpenCode auth is available (once per process).""" - global _auth_checked # noqa: PLW0603 - if _auth_checked: - return - _check_binary_compat() - if _has_opencode_auth(): - _auth_checked = True - return - raise OpenCodeAuthError() - - -def _check_binary_compat() -> None: - """Warn if the opencode binary is v0.x (archived Go version). - - v1.x (anomalyco/opencode) outputs version strings like "1.18.14". - v0.x (opencode-ai/opencode) outputs "opencode version v0.x.x". - """ - global _compat_checked # noqa: PLW0603 - if _compat_checked: - return - _compat_checked = True - - import re - import shutil - - if not shutil.which("opencode"): - return - - try: - result = subprocess.run( - ["opencode", "--version"], - capture_output=True, - text=True, - timeout=10, - ) - output = (getattr(result, "stdout", None) or "").strip() + ( - getattr(result, "stderr", None) or "" - ).strip() - if re.search(r"\bv?0\.\d+\.\d+", output): - log.warning( - "opencode_binary_v0x_detected", - output=output, - hint=( - "The opencode binary appears to be v0.x (archived). " - "The factory requires OpenCode v1.x (anomalyco/opencode). " - "Install via: curl -fsSL https://opencode.ai/install | bash " - "or: npm i -g opencode-ai" - ), - ) - return - log.debug("opencode_binary_compat_ok", output=output) - except FileNotFoundError: - pass - except subprocess.TimeoutExpired: - log.debug("opencode_version_check_timeout") - - -def is_opencode_dry_run() -> bool: - """Return True if OpenCode dry-run mode is enabled.""" - from factory.user_config import resolve - - val = resolve("opencode_dry_run", env_var="FACTORY_OPENCODE_DRY_RUN") or "" - return val.lower() in ("1", "true", "yes") - - -class OpenCodeRunner: - """Runner implementation for OpenCode v1.x CLI (anomalyco/opencode).""" - - name: str = "opencode" - - def __init__( - self, - cycle_start: datetime | None = None, - project_path: Path | None = None, - ) -> None: - if cycle_start is not None: - self.cycle_start = cycle_start - elif project_path is not None: - from factory.ceo_completion import read_cycle_state - - state = read_cycle_state(project_path) - self.cycle_start = state.started_at if state else datetime.now(timezone.utc) - else: - self.cycle_start = datetime.now(timezone.utc) - self._role: str = "unknown" - - @classmethod - def metadata(cls) -> RunnerMeta: - from factory.runners.protocol import RunnerMeta - - return RunnerMeta( - name="opencode", - display_name="OpenCode", - binary="opencode", - install_hint="curl -fsSL https://opencode.ai/install | bash", - required_env_vars=[], - supports_model_override=True, - supports_interactive=True, - supports_streaming=True, - supports_usage_telemetry=False, - supports_session_name=True, - supports_session_resume=True, - supports_background=False, - custom_auth_check=_has_opencode_auth, - ) - - def build_command( - self, request: AgentRunRequest - ) -> tuple[list[str], dict[str, str], list[Path]]: - """Build the OpenCode v1.x CLI command for headless execution.""" - cwd = Path(request.cwd) - agents_md_path = cwd / "AGENTS.md" - agents_md_path.write_text(request.prompt) - temp_files: list[Path] = [agents_md_path] - - cmd = ["opencode", "run", request.task, "--format", "json", "--dir", str(request.cwd)] - - if request.skip_permissions: - cmd.append("--auto") - - if request.model: - cmd.extend(["--model", request.model]) - - if request.session_name: - cmd.extend(["--title", request.session_name]) - - if request.resume_session_id: - cmd.extend(["--session", request.resume_session_id]) - - if request.session_id: - cmd.append("--continue") - - env = {k: v for k, v in os.environ.items() if k != "VIRTUAL_ENV"} - return cmd, env, temp_files - - def build_interactive_command( - self, request: AgentRunRequest - ) -> tuple[list[str], dict[str, str], list[Path]]: - """Build the CLI command for interactive (TUI) mode.""" - cwd = Path(request.cwd) - agents_md_path = cwd / "AGENTS.md" - agents_md_path.write_text(request.prompt) - temp_files: list[Path] = [agents_md_path] - - cmd = ["opencode", "--prompt", request.task] - - if request.skip_permissions: - cmd.append("--auto") - - if request.model: - cmd.extend(["--model", request.model]) - - if request.resume_session_id: - cmd.extend(["--session", request.resume_session_id]) - - cmd.append(str(request.cwd)) - - env = {k: v for k, v in os.environ.items() if k != "VIRTUAL_ENV"} - return cmd, env, temp_files - - async def headless(self, request: AgentRunRequest) -> AgentRunResult: - """Run a headless OpenCode v1.x invocation.""" - from factory.models import AgentRunResult - - tmux_persist = request.extras.get("tmux_persist", False) - if tmux_persist: - return AgentRunResult( - stdout="Error: --tmux-persist is not supported with the opencode runner. Use --runner claude.", - return_code=1, - ) - - background = request.extras.get("background", False) - if background: - return AgentRunResult( - stdout="Error: --bg is not supported with the opencode runner. Use --runner claude.", - return_code=1, - ) - - self._role = request.role - project_path = request.project_path or self._find_project_path(request.cwd) - - if is_opencode_dry_run(): - from factory.runners._subprocess import make_dry_run_result - - result = make_dry_run_result("opencode", request.role, request.cwd, request.task) - log_usage( - project_path, - request.role, - request.cwd, - 0.0, - 0, - dry_run=True, - runner_name=_RUNNER_NAME, - ) - return result - - _check_auth() - - try: - check_ceilings(project_path, self.cycle_start, runner_name=_RUNNER_NAME) - except CeilingExceededError as e: - self._emit_ceiling_event(project_path, e) - return AgentRunResult(stdout=str(e), return_code=1) - - cmd, env, temp_files = self.build_command(request) - - log.info("opencode_headless", cwd=str(request.cwd), role=request.role, model=request.model) - - start_time = time.monotonic() - - try: - result = await run_subprocess( - cmd, - cwd=str(request.cwd), - env=env, - timeout=request.timeout, - runner_name="opencode", - role=request.role, - sanitize=True, - ) - - duration = time.monotonic() - start_time - log_usage( - project_path, - request.role, - request.cwd, - duration, - result.return_code, - dry_run=False, - runner_name=_RUNNER_NAME, - ) - - return result - finally: - for f in temp_files: - f.unlink(missing_ok=True) - - def interactive_run(self, request: AgentRunRequest) -> int: - """Run an interactive OpenCode v1.x session as a subprocess.""" - project_path = request.project_path or self._find_project_path(request.cwd) - - if is_opencode_dry_run(): - print("[DRY-RUN] Would exec: opencode (interactive)") - print(f"[DRY-RUN] Task: {request.task[:200]}...") - return 0 - - _check_auth() - - try: - check_ceilings(project_path, self.cycle_start, runner_name=_RUNNER_NAME) - except CeilingExceededError as e: - print(f"ERROR: {e}") - return 1 - - cmd, env, temp_files = self.build_interactive_command(request) - - log.info("opencode_interactive", cwd=str(request.cwd)) - - try: - result = subprocess.run(cmd, cwd=request.cwd, env=env) - return result.returncode - finally: - for f in temp_files: - f.unlink(missing_ok=True) - - def _find_project_path(self, cwd: Path) -> Path: - """Find the project root (directory containing .factory/).""" - path = cwd.resolve() - while path != path.parent: - if (path / ".factory").is_dir(): - return path - path = path.parent - return cwd.resolve() - - def _emit_ceiling_event(self, project_path: Path, error: CeilingExceededError) -> None: - """Emit a structured event when a ceiling is hit.""" - try: - from factory.events import emit_event - - emit_event( - project_path, - "opencode.ceiling_exceeded", - data={ - "ceiling": error.ceiling_name, - "current": error.current, - "limit": error.limit, - "env_var": error.env_var, - }, - ) - except Exception: - log.debug("opencode_ceiling_event_failed", exc_info=True) diff --git a/factory/runners/usage.py b/factory/runners/usage.py deleted file mode 100644 index 5d034e5d5..000000000 --- a/factory/runners/usage.py +++ /dev/null @@ -1,179 +0,0 @@ -"""Runner usage tracking — log and ceiling enforcement. - -Generalized for any runner (Bob, OpenCode, etc.) via runner_name parameter. -""" - -from __future__ import annotations - -import json -from dataclasses import dataclass -from datetime import datetime, timezone -from pathlib import Path -from typing import TypedDict - -import structlog - -log = structlog.get_logger() - - -class UsageEntry(TypedDict): - timestamp: str - role: str - cwd: str - duration_seconds: float - exit_code: int - dry_run: bool - - -def get_usage_log_path(project_path: Path, runner_name: str = "bob") -> Path: - """Return the path to the usage log for a project.""" - return project_path / ".factory" / f"{runner_name}_usage.jsonl" - - -def log_usage( - project_path: Path, - role: str, - cwd: Path, - duration_seconds: float, - exit_code: int, - dry_run: bool = False, - runner_name: str = "bob", -) -> None: - """Append a usage entry to the project's usage log.""" - log_path = get_usage_log_path(project_path, runner_name) - log_path.parent.mkdir(parents=True, exist_ok=True) - - entry: UsageEntry = { - "timestamp": datetime.now(timezone.utc).isoformat(), - "role": role, - "cwd": str(cwd), - "duration_seconds": duration_seconds, - "exit_code": exit_code, - "dry_run": dry_run, - } - - with open(log_path, "a") as f: - f.write(json.dumps(entry) + "\n") - - -def count_cycle_invocations( - project_path: Path, - cycle_start: datetime | None = None, - runner_name: str = "bob", -) -> int: - """Count non-dry-run invocations in the current cycle. - - If cycle_start is None, returns 0 (no cycle tracking without explicit start). - """ - if cycle_start is None: - return 0 - - log_path = get_usage_log_path(project_path, runner_name) - if not log_path.exists(): - return 0 - - count = 0 - cycle_start_iso = cycle_start.isoformat() - - with open(log_path) as f: - for line in f: - line = line.strip() - if not line: - continue - try: - entry = json.loads(line) - ts = entry.get("timestamp", "") - if ts >= cycle_start_iso and not entry.get("dry_run", False): - count += 1 - except json.JSONDecodeError: - continue - - return count - - -def get_cycle_ceiling(runner_name: str = "bob") -> int: - """Get the per-cycle invocation ceiling from env var.""" - from factory.user_config import resolve - - upper = runner_name.upper() - env_var = f"FACTORY_{upper}_MAX_INVOCATIONS_PER_CYCLE" - config_key = f"{runner_name}_max_invocations_per_cycle" - return int(resolve(config_key, env_var=env_var, default="8") or "8") - - -class CeilingExceededError(Exception): - """Raised when a runner invocation ceiling is exceeded.""" - - def __init__(self, ceiling_name: str, current: int, limit: int, env_var: str, runner_name: str = "bob") -> None: - self.ceiling_name = ceiling_name - self.current = current - self.limit = limit - self.env_var = env_var - self.runner_name = runner_name - display = runner_name.capitalize() - super().__init__( - f"{display} {ceiling_name} ceiling exceeded: {current}/{limit}. " - f"To increase, set {env_var}={limit + 5}" - ) - - -@dataclass -class CeilingWarning: - """Warning when approaching a ceiling (≤2 invocations remaining).""" - - ceiling_name: str - remaining: int - limit: int - - -def _emit_warning_event(project_path: Path, warning: CeilingWarning, runner_name: str = "bob") -> None: - """Emit a warning event to .factory/events.jsonl.""" - try: - from factory.events import emit_event - - emit_event( - project_path, - f"{runner_name}.ceiling_warning", - data={ - "ceiling": warning.ceiling_name, - "remaining": warning.remaining, - "limit": warning.limit, - }, - ) - except Exception: - log.warning("Failed to emit ceiling warning event", exc_info=True) - - -def check_ceilings( - project_path: Path, - cycle_start: datetime | None = None, - runner_name: str = "bob", -) -> CeilingWarning | None: - """Check per-cycle ceiling before a runner invocation. - - Raises CeilingExceededError if the per-cycle ceiling is exceeded. - Returns CeilingWarning if ≤2 invocations remain before the ceiling. - """ - upper = runner_name.upper() - env_var = f"FACTORY_{upper}_MAX_INVOCATIONS_PER_CYCLE" - - cycle_count = count_cycle_invocations(project_path, cycle_start, runner_name) - cycle_limit = get_cycle_ceiling(runner_name) - if cycle_count >= cycle_limit: - raise CeilingExceededError( - "per-cycle", cycle_count, cycle_limit, env_var, runner_name - ) - - remaining = cycle_limit - cycle_count - if remaining <= 2: - warning = CeilingWarning("per-cycle", remaining, cycle_limit) - log.warning( - f"{runner_name}_ceiling_approaching", - ceiling=warning.ceiling_name, - remaining=warning.remaining, - limit=warning.limit, - ) - _emit_warning_event(project_path, warning, runner_name) - return warning - - return None diff --git a/factory/user_config.py b/factory/user_config.py index f59b7d7ed..773ae253f 100644 --- a/factory/user_config.py +++ b/factory/user_config.py @@ -39,7 +39,7 @@ # See: factory config show [defaults] -# runner = "claude" # CLI backend: "claude", "bob", or "codex" +# runner = "claude" # CLI backend (claude is the only built-in runner) # model = "" # Claude model for agent subprocesses # projects_dir = "~/factory-projects" # Root for factory-managed projects # tmux_persist = false # Launch agents in tmux windows @@ -51,14 +51,6 @@ # FACTORY_RUNNER = "claude" # ANTHROPIC_API_KEY = "sk-ant-..." # -# [credentials.bob] -# FACTORY_RUNNER = "bob" -# BOBSHELL_API_KEY = "..." -# -# [credentials.codex] -# FACTORY_RUNNER = "codex" -# CODEX_API_KEY = "..." -# # [credentials.litellm-proxy] # FACTORY_RUNNER = "claude" # FACTORY_MODEL = "your-model-name" @@ -310,8 +302,6 @@ def migrate_env_to_config() -> str: "FACTORY_REGISTRY_DIR": "registry_dir", "FACTORY_MANAGED_DIRS": "managed_dirs", "FACTORY_RUNNER_QUIET": "runner_quiet", - "FACTORY_BOB_DRY_RUN": "bob_dry_run", - "FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE": "bob_max_invocations_per_cycle", "FACTORY_CEO_RESPAWN_DISABLED": "ceo_respawn_disabled", "FACTORY_CEO_MAX_RESPAWNS": "ceo_max_respawns", "FACTORY_REMOVE_WORKTREE": "remove_worktree", diff --git a/scripts/sync_agents.py b/scripts/sync_agents.py index e8049e1a2..f66b8d5a4 100644 --- a/scripts/sync_agents.py +++ b/scripts/sync_agents.py @@ -16,14 +16,11 @@ from factory.agents.plugin import ( check_agents_in_sync, - check_codex_agents_in_sync, generate_agent_content, - generate_codex_agent_toml, load_agent_config, ) _AGENTS_DIR = Path(__file__).resolve().parent.parent / "agents" -_CODEX_AGENTS_DIR = Path(__file__).resolve().parent.parent / "codex-agents" def main() -> int: @@ -31,17 +28,14 @@ def main() -> int: if check_mode: out_of_sync = check_agents_in_sync(_AGENTS_DIR) - codex_out_of_sync = check_codex_agents_in_sync(_CODEX_AGENTS_DIR) - all_issues = out_of_sync + [f"{r} (codex)" for r in codex_out_of_sync] - if all_issues: - print(f"Out of sync: {', '.join(all_issues)}", file=sys.stderr) + if out_of_sync: + print(f"Out of sync: {', '.join(out_of_sync)}", file=sys.stderr) print("Run: python scripts/sync_agents.py", file=sys.stderr) return 1 print("All plugin agents are in sync.") return 0 _AGENTS_DIR.mkdir(exist_ok=True) - _CODEX_AGENTS_DIR.mkdir(exist_ok=True) config = load_agent_config() for role in config: @@ -50,14 +44,7 @@ def main() -> int: out_path.write_text(content) print(f" {role} -> {out_path}") - for role in config: - toml_content = generate_codex_agent_toml(role) - toml_path = _CODEX_AGENTS_DIR / f"{role}.toml" - toml_path.write_text(toml_content) - print(f" {role} -> {toml_path}") - print(f"\nGenerated {len(config)} agent files in {_AGENTS_DIR} (Markdown)") - print(f"Generated {len(config)} agent files in {_CODEX_AGENTS_DIR} (TOML)") return 0 diff --git a/skills/study/SKILL.md b/skills/study/SKILL.md deleted file mode 100644 index b28404b9f..000000000 --- a/skills/study/SKILL.md +++ /dev/null @@ -1,57 +0,0 @@ ---- -name: study -description: "Analyze the current codebase using Factory's observation engine and code graph. Generates a report covering code quality, eval scores, structural analysis, open issues, backlog items, observability coverage, and improvement opportunities. Use when the user wants to understand the state of their project before making changes." -disable-model-invocation: true ---- - -# /factory:study - -Analyze the current codebase and generate an observation report with structural graph analysis. - -## Prerequisites - -```bash -command -v factory >/dev/null 2>&1 || uv tool install "${CLAUDE_PLUGIN_ROOT}" -``` - -## Execution - -```bash -factory graph update "$(pwd)" -factory study "$(pwd)" -``` - -Check whether a code knowledge graph is available by running `factory graph status "$(pwd)"`. - -If the graph is available (status shows node/edge counts), explore the code graph: - -```bash -factory graph query "$(pwd)" "" --depth 2 -factory graph explain "$(pwd)" "" -factory graph path "$(pwd)" "" "" -``` - -Write graph findings to `.factory/strategy/graph-context.md`, then combine: - -```bash -cat .factory/strategy/observations.md .factory/strategy/graph-context.md \ - > .factory/strategy/study-combined.md -``` - -The combined report at `.factory/strategy/study-combined.md` covers: - -- **Eval scores** — current composite and per-dimension breakdown -- **Open issues** — from GitHub, if available -- **Backlog items** — pending work from `.factory/strategy/backlog.md` -- **Observability coverage** — logging density and uninstrumented files -- **Hypothesis budget** — how many improvements to target this cycle -- **Cross-project insights** — patterns from sibling projects (if any) -- **Structural analysis** — key modules, dependency paths, architectural layers, entry points - -For cross-project insights, pass `--projects-dir`: - -```bash -factory study "$(pwd)" --projects-dir ~/factory-projects -``` - -After studying, use `/factory:implement` to act on the findings. diff --git a/tests/conftest.py b/tests/conftest.py index 8217e6fd8..057997958 100644 --- a/tests/conftest.py +++ b/tests/conftest.py @@ -10,10 +10,6 @@ from factory.models import FactoryConfig -# CRITICAL: Set FACTORY_BOB_DRY_RUN=1 before any tests run. -# This ensures BobRunner never invokes real bob during tests. -os.environ["FACTORY_BOB_DRY_RUN"] = "1" - # Disable CEO completion guard by default in tests. # Tests that need to exercise the guard can unset this or test with mocked invoke_agent. os.environ["FACTORY_CEO_RESPAWN_DISABLED"] = "1" diff --git a/tests/test_agents.py b/tests/test_agents.py index 257144906..7974d4c5e 100644 --- a/tests/test_agents.py +++ b/tests/test_agents.py @@ -391,7 +391,7 @@ def test_error_message_is_actionable(self): assert "2 consecutive" in msg assert "researcher" in msg assert "events.jsonl" in msg - assert "BOBSHELL_API_KEY" in msg # hint about the common cause + assert "ANTHROPIC_API_KEY" in msg # hint about the common cause class TestCeoPromptNoBackgroundSpawning: @@ -507,16 +507,6 @@ def test_supports_background_on_runner_meta(self): assert ClaudeRunner.metadata().supports_background is True - def test_other_runners_no_background(self): - """Non-claude runners have supports_background=False.""" - from factory.runners.bob import BobRunner - from factory.runners.codex import CodexRunner - from factory.runners.opencode import OpenCodeRunner - - assert BobRunner.metadata().supports_background is False - assert CodexRunner.metadata().supports_background is False - assert OpenCodeRunner.metadata().supports_background is False - def test_resolve_background_flag(self, monkeypatch): """_resolve_background resolves CLI flag correctly.""" import argparse diff --git a/tests/test_ceo_completion.py b/tests/test_ceo_completion.py index 20601aa8f..6eb7f207d 100644 --- a/tests/test_ceo_completion.py +++ b/tests/test_ceo_completion.py @@ -107,18 +107,7 @@ def test_cycle_state_truncates_long_prompt(self, tmp_path: Path) -> None: class TestBudgetAllowsRespawn: - """Tests for _budget_allows_respawn(). - - With only per-cycle limits (no daily/session limit), respawn is always allowed. - Per-cycle limits are enforced within BobRunner during execution. - """ - - def test_bob_always_allowed(self, tmp_path: Path) -> None: - from factory.ceo_completion import _budget_allows_respawn - - (tmp_path / ".factory").mkdir() - # Always returns True - per-cycle limits are enforced within BobRunner - assert _budget_allows_respawn("bob", tmp_path) is True + """Tests for _budget_allows_respawn().""" def test_claude_always_allowed(self, tmp_path: Path) -> None: from factory.ceo_completion import _budget_allows_respawn diff --git a/tests/test_codex_runner.py b/tests/test_codex_runner.py deleted file mode 100644 index 74760d750..000000000 --- a/tests/test_codex_runner.py +++ /dev/null @@ -1,612 +0,0 @@ -"""Tests for factory/runners/codex.py — CodexRunner implementation.""" - -from pathlib import Path -from unittest.mock import AsyncMock, patch - -import pytest - -import factory.runners.codex as codex_module -from factory.models import AgentRunRequest, AgentRunResult -from factory.runners import CodexRunner, get_runner, is_codex_dry_run -from factory.runners.codex import CodexAuthError, _check_auth - - -@pytest.fixture(autouse=True) -def _reset_codex_auth() -> None: - codex_module._auth_checked = False - - -class TestGetRunnerCodex: - def test_explicit_codex(self) -> None: - runner = get_runner("codex") - assert runner.name == "codex" - - def test_from_env_var(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("FACTORY_RUNNER", "codex") - runner = get_runner() - assert runner.name == "codex" - - def test_explicit_overrides_env(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("FACTORY_RUNNER", "codex") - runner = get_runner("claude") - assert runner.name == "claude" - - -class TestCodexDryRun: - def test_dry_run_true(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("FACTORY_CODEX_DRY_RUN", "1") - assert is_codex_dry_run() is True - - def test_dry_run_false(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - assert is_codex_dry_run() is False - - def test_dry_run_true_word(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("FACTORY_CODEX_DRY_RUN", "true") - assert is_codex_dry_run() is True - - async def test_headless_dry_run_returns_stub( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("FACTORY_CODEX_DRY_RUN", "1") - - runner = CodexRunner() - result = await runner.headless( - AgentRunRequest( - prompt="You are a test agent.", - task="Say hello", - cwd=tmp_path, - role="researcher", - ) - ) - - assert result.return_code == 0 - assert "[DRY-RUN]" in result.stdout - assert "researcher" in result.stdout - assert result.usage is None - - def test_interactive_run_dry_run( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] - ) -> None: - monkeypatch.setenv("FACTORY_CODEX_DRY_RUN", "1") - - runner = CodexRunner() - code = runner.interactive_run( - AgentRunRequest( - prompt="Test prompt", - task="Test task", - cwd=tmp_path, - role="ceo", - ) - ) - - assert code == 0 - captured = capsys.readouterr() - assert "[DRY-RUN]" in captured.out - - -class TestCodexAuth: - def test_auth_fails_without_key(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.delenv("CODEX_API_KEY", raising=False) - monkeypatch.delenv("OPENAI_API_KEY", raising=False) - - with patch("factory.runners.codex._has_codex_oauth", return_value=False): - with pytest.raises(CodexAuthError, match="CODEX_API_KEY"): - _check_auth() - - def test_auth_passes_with_codex_key(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("OPENAI_API_KEY", raising=False) - - _check_auth() - assert codex_module._auth_checked is True - - def test_auth_passes_with_openai_key(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.delenv("CODEX_API_KEY", raising=False) - monkeypatch.setenv("OPENAI_API_KEY", "test-openai-key") - - with patch("factory.runners.codex._has_codex_oauth", return_value=False): - _check_auth() - assert codex_module._auth_checked is True - - def test_auth_prefers_oauth_over_api_key(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("OPENAI_API_KEY", "test-key") - - with patch("factory.runners.codex._has_codex_oauth", return_value=True): - _check_auth() - assert codex_module._auth_checked is True - - async def test_headless_fails_without_key( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.delenv("CODEX_API_KEY", raising=False) - monkeypatch.delenv("OPENAI_API_KEY", raising=False) - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - - runner = CodexRunner() - with patch("factory.runners.codex._has_codex_oauth", return_value=False): - with pytest.raises(CodexAuthError): - await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="researcher", - ) - ) - - -class TestCodexEnvMapping: - def test_codex_key_mapped_to_openai(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("CODEX_API_KEY", "my-codex-key") - monkeypatch.delenv("OPENAI_API_KEY", raising=False) - - from factory.runners.codex import _make_codex_env - - with patch("factory.runners.codex._has_codex_oauth", return_value=False): - env, tmpdir = _make_codex_env() - tmpdir.cleanup() - assert env["OPENAI_API_KEY"] == "my-codex-key" - assert "VIRTUAL_ENV" not in env - - def test_openai_key_not_overridden_without_oauth(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("CODEX_API_KEY", "codex-key") - monkeypatch.setenv("OPENAI_API_KEY", "openai-key") - - from factory.runners.codex import _make_codex_env - - with patch("factory.runners.codex._has_codex_oauth", return_value=False): - env, tmpdir = _make_codex_env() - tmpdir.cleanup() - assert env["OPENAI_API_KEY"] == "openai-key" - - def test_oauth_strips_api_keys(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("OPENAI_API_KEY", "openai-key") - monkeypatch.setenv("CODEX_API_KEY", "codex-key") - - from factory.runners.codex import _make_codex_env - - with patch("factory.runners.codex._has_codex_oauth", return_value=True): - env, tmpdir = _make_codex_env() - assert tmpdir is None - assert "OPENAI_API_KEY" not in env - assert "CODEX_API_KEY" not in env - - def test_virtual_env_stripped(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("VIRTUAL_ENV", "/some/venv") - - from factory.runners.codex import _make_codex_env - - with patch("factory.runners.codex._has_codex_oauth", return_value=False): - env, tmpdir = _make_codex_env() - if tmpdir is not None: - tmpdir.cleanup() - assert "VIRTUAL_ENV" not in env - - -class TestCodexHeadless: - async def test_builds_correct_command( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - - runner = CodexRunner() - - with patch( - "factory.runners.codex.run_subprocess", new_callable=AsyncMock - ) as mock_run: - mock_run.return_value = AgentRunResult(stdout="output", return_code=0) - - result = await runner.headless( - AgentRunRequest( - prompt="You are a test agent.", - task="Say hello", - cwd=tmp_path, - timeout=60.0, - model="gpt-5.4", - ) - ) - - assert result.return_code == 0 - assert result.stdout == "output" - assert result.usage is None - - call_args = mock_run.call_args - cmd = call_args[0][0] - assert cmd[0] == "codex" - assert cmd[1] == "exec" - assert "--ignore-user-config" in cmd - assert "--sandbox" in cmd - assert "workspace-write" in cmd - assert "--ask-for-approval" not in cmd - assert "--model" in cmd - assert "gpt-5.4" in cmd - assert "--skip-git-repo-check" in cmd - assert "--" in cmd - - async def test_combines_prompt_and_task( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - - runner = CodexRunner() - - with patch( - "factory.runners.codex.run_subprocess", new_callable=AsyncMock - ) as mock_run: - mock_run.return_value = AgentRunResult(stdout="ok", return_code=0) - - await runner.headless( - AgentRunRequest( - prompt="You are the CEO.", - task="Run the experiment", - cwd=tmp_path, - ) - ) - - cmd = mock_run.call_args[0][0] - dash_idx = cmd.index("--") - full_prompt = cmd[dash_idx + 1] - assert "You are the CEO." in full_prompt - assert "Run the experiment" in full_prompt - assert "## Current Task" in full_prompt - - async def test_no_sandbox_flags_when_permissions_not_skipped( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - - runner = CodexRunner() - - with patch( - "factory.runners.codex.run_subprocess", new_callable=AsyncMock - ) as mock_run: - mock_run.return_value = AgentRunResult(stdout="ok", return_code=0) - - await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - skip_permissions=False, - ) - ) - - cmd = mock_run.call_args[0][0] - assert "--sandbox" not in cmd - assert "--ask-for-approval" not in cmd - - async def test_no_model_flag_when_none( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - - runner = CodexRunner() - - with patch( - "factory.runners.codex.run_subprocess", new_callable=AsyncMock - ) as mock_run: - mock_run.return_value = AgentRunResult(stdout="ok", return_code=0) - - await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - model=None, - ) - ) - - cmd = mock_run.call_args[0][0] - assert "--model" not in cmd - - async def test_handles_timeout( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - - with patch( - "factory.runners.codex.run_subprocess", new_callable=AsyncMock - ) as mock_run: - mock_run.return_value = AgentRunResult( - stdout="Agent timed out after 0.1s", return_code=1 - ) - - runner = CodexRunner() - result = await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="researcher", - timeout=0.1, - ) - ) - - assert result.return_code == 1 - assert "timed out" in result.stdout.lower() - assert result.usage is None - - async def test_handles_missing_binary( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - - with patch( - "factory.runners.codex.run_subprocess", - new_callable=AsyncMock, - ) as mock_run: - mock_run.return_value = AgentRunResult( - stdout="Error: 'codex' CLI not found on PATH", return_code=1 - ) - - runner = CodexRunner() - result = await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - ) - ) - - assert result.return_code == 1 - assert "not found" in result.stdout.lower() - assert result.usage is None - - async def test_passes_env_with_openai_key( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("OPENAI_API_KEY", raising=False) - monkeypatch.setenv("VIRTUAL_ENV", "/some/venv") - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - - runner = CodexRunner() - - with patch("factory.runners.codex._has_codex_oauth", return_value=False): - with patch( - "factory.runners.codex.run_subprocess", new_callable=AsyncMock - ) as mock_run: - mock_run.return_value = AgentRunResult(stdout="ok", return_code=0) - - await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - ) - ) - - call_kwargs = mock_run.call_args.kwargs - assert "VIRTUAL_ENV" not in call_kwargs["env"] - assert call_kwargs["env"]["OPENAI_API_KEY"] == "test-key" - - -class TestCodexStreaming: - async def test_uses_streaming_prefix( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - monkeypatch.delenv("FACTORY_RUNNER_QUIET", raising=False) - - runner = CodexRunner() - - with patch( - "factory.runners.codex.run_subprocess", new_callable=AsyncMock - ) as mock_run: - mock_run.return_value = AgentRunResult(stdout="output\n", return_code=0) - - await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="builder", - ) - ) - - mock_run.assert_called_once() - call_kwargs = mock_run.call_args.kwargs - assert call_kwargs["runner_name"] == "codex" - assert call_kwargs["role"] == "builder" - - async def test_codex_runner_does_not_sanitize( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """CodexRunner.headless() does not sanitize (default False) — issue #379.""" - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - monkeypatch.delenv("FACTORY_RUNNER_QUIET", raising=False) - - runner = CodexRunner() - - with patch( - "factory.runners.codex.run_subprocess", new_callable=AsyncMock - ) as mock_run: - mock_run.return_value = AgentRunResult(stdout="output\n", return_code=0) - - await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="builder", - ) - ) - - mock_run.assert_called_once() - # run_subprocess defaults sanitize=False; CodexRunner does not pass it - assert mock_run.call_args.kwargs.get("sanitize", False) is False - - -class TestCodexBuildInteractiveCommand: - """Tests for CodexRunner.build_interactive_command().""" - - def test_base_command_structure( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - runner = CodexRunner() - - with patch("factory.runners.codex._has_codex_oauth", return_value=False): - cmd, env, temp_files = runner.build_interactive_command(AgentRunRequest( - prompt="You are the CEO.", - task="Start session", - cwd=tmp_path, - model="gpt-5.4", - skip_permissions=True, - )) - - assert cmd[0] == "codex" - full_prompt = cmd[1] - assert "You are the CEO." in full_prompt - assert "Start session" in full_prompt - assert "## Current Task" in full_prompt - assert "exec" not in cmd - assert "--" not in cmd - assert "--skip-git-repo-check" not in cmd - assert "--ignore-user-config" in cmd - assert "--full-auto" in cmd - assert "--model" in cmd - assert "gpt-5.4" in cmd - assert temp_files == [] - assert "VIRTUAL_ENV" not in env - - if hasattr(runner, "_tmpdir") and runner._tmpdir is not None: - runner._tmpdir.cleanup() - - def test_no_permission_flags_without_skip( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - runner = CodexRunner() - - with patch("factory.runners.codex._has_codex_oauth", return_value=False): - cmd, _, _ = runner.build_interactive_command(AgentRunRequest( - prompt="Test", task="Test", cwd=tmp_path, skip_permissions=False, - )) - - assert "--full-auto" not in cmd - assert "--sandbox" not in cmd - - if hasattr(runner, "_tmpdir") and runner._tmpdir is not None: - runner._tmpdir.cleanup() - - def test_no_model_flag_when_none( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - runner = CodexRunner() - - with patch("factory.runners.codex._has_codex_oauth", return_value=False): - cmd, _, _ = runner.build_interactive_command(AgentRunRequest( - prompt="Test", task="Test", cwd=tmp_path, model=None, - )) - - assert "--model" not in cmd - - if hasattr(runner, "_tmpdir") and runner._tmpdir is not None: - runner._tmpdir.cleanup() - - def test_env_from_make_codex_env( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("OPENAI_API_KEY", raising=False) - monkeypatch.setenv("VIRTUAL_ENV", "/some/venv") - runner = CodexRunner() - - with patch("factory.runners.codex._has_codex_oauth", return_value=False): - _, env, _ = runner.build_interactive_command(AgentRunRequest( - prompt="Test", task="Test", cwd=tmp_path, - )) - - assert "VIRTUAL_ENV" not in env - assert env["OPENAI_API_KEY"] == "test-key" - - if hasattr(runner, "_tmpdir") and runner._tmpdir is not None: - runner._tmpdir.cleanup() - - -class TestCodexInteractive: - def test_interactive_run_builds_correct_command( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - - runner = CodexRunner() - - with patch("subprocess.run") as mock_run: - mock_run.return_value = type("Result", (), {"returncode": 0})() - code = runner.interactive_run( - AgentRunRequest( - prompt="You are the CEO.", - task="Start session", - cwd=tmp_path, - model="gpt-5.4", - skip_permissions=True, - ) - ) - - assert code == 0 - cmd = mock_run.call_args[0][0] - assert cmd[0] == "codex" - assert "--ignore-user-config" in cmd - assert "--full-auto" in cmd - assert "--model" in cmd - assert "gpt-5.4" in cmd - - def test_interactive_run_no_sandbox_without_skip( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - - runner = CodexRunner() - - with patch("subprocess.run") as mock_run: - mock_run.return_value = type("Result", (), {"returncode": 0})() - runner.interactive_run( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - skip_permissions=False, - ) - ) - - cmd = mock_run.call_args[0][0] - assert "--full-auto" not in cmd - - def test_interactive_run_passes_env( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("CODEX_API_KEY", "test-key") - monkeypatch.delenv("OPENAI_API_KEY", raising=False) - monkeypatch.setenv("VIRTUAL_ENV", "/some/venv") - monkeypatch.delenv("FACTORY_CODEX_DRY_RUN", raising=False) - - runner = CodexRunner() - - with patch("factory.runners.codex._has_codex_oauth", return_value=False): - with patch("subprocess.run") as mock_run: - mock_run.return_value = type("Result", (), {"returncode": 0})() - runner.interactive_run( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - ) - ) - - call_kwargs = mock_run.call_args.kwargs - assert "VIRTUAL_ENV" not in call_kwargs["env"] - assert call_kwargs["env"]["OPENAI_API_KEY"] == "test-key" diff --git a/tests/test_opencode_runner.py b/tests/test_opencode_runner.py deleted file mode 100644 index 661bb535b..000000000 --- a/tests/test_opencode_runner.py +++ /dev/null @@ -1,646 +0,0 @@ -"""Tests for factory/runners/opencode.py — OpenCode v1.x runner.""" - -from __future__ import annotations - -import json -from datetime import datetime, timezone -from pathlib import Path -from unittest.mock import AsyncMock, MagicMock, patch - -import pytest - -import factory.runners.opencode as oc_module -from factory.models import AgentRunRequest, AgentRunResult -from factory.runners.opencode import ( - OpenCodeAuthError, - OpenCodeRunner, - _check_auth, - _check_binary_compat, - _has_opencode_auth, - is_opencode_dry_run, -) - - -@pytest.fixture(autouse=True) -def _reset_opencode_globals() -> None: - """Reset module-level auth/compat guards before each test.""" - oc_module._auth_checked = False - oc_module._compat_checked = False - - -# --------------------------------------------------------------------------- -# OpenCodeAuthError -# --------------------------------------------------------------------------- - - -class TestOpenCodeAuthError: - def test_error_message(self) -> None: - err = OpenCodeAuthError() - assert "opencode auth login" in str(err) - assert "ANTHROPIC_API_KEY" in str(err) - assert "config.toml" in str(err) - - -# --------------------------------------------------------------------------- -# _has_opencode_auth -# --------------------------------------------------------------------------- - - -class TestHasOpenCodeAuth: - def test_true_with_opencode_dir(self, tmp_path: Path) -> None: - with patch("factory.runners.opencode.Path.home", return_value=tmp_path): - (tmp_path / ".opencode").mkdir() - assert _has_opencode_auth() is True - - def test_true_with_anthropic_key(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("ANTHROPIC_API_KEY", "sk-ant-test") - with patch("factory.runners.opencode.Path.home", return_value=Path("/nonexistent")): - assert _has_opencode_auth() is True - - def test_true_with_openai_key(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("OPENAI_API_KEY", "sk-test") - with patch("factory.runners.opencode.Path.home", return_value=Path("/nonexistent")): - assert _has_opencode_auth() is True - - def test_false_without_anything(self, monkeypatch: pytest.MonkeyPatch) -> None: - for var in oc_module._PROVIDER_ENV_VARS: - monkeypatch.delenv(var, raising=False) - with patch("factory.runners.opencode.Path.home", return_value=Path("/nonexistent")): - assert _has_opencode_auth() is False - - -# --------------------------------------------------------------------------- -# _check_auth -# --------------------------------------------------------------------------- - - -class TestCheckAuth: - def test_skips_when_already_checked(self) -> None: - oc_module._auth_checked = True - _check_auth() - - def test_passes_with_opencode_dir(self, tmp_path: Path) -> None: - with patch("factory.runners.opencode._check_binary_compat"): - with patch("factory.runners.opencode.Path.home", return_value=tmp_path): - (tmp_path / ".opencode").mkdir() - _check_auth() - assert oc_module._auth_checked is True - - def test_passes_with_env_key(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("ANTHROPIC_API_KEY", "test-key") - with patch("factory.runners.opencode._check_binary_compat"): - _check_auth() - assert oc_module._auth_checked is True - - def test_raises_without_auth(self, monkeypatch: pytest.MonkeyPatch) -> None: - for var in oc_module._PROVIDER_ENV_VARS: - monkeypatch.delenv(var, raising=False) - with patch("factory.runners.opencode._check_binary_compat"): - with patch("factory.runners.opencode.Path.home", return_value=Path("/nonexistent")): - with pytest.raises(OpenCodeAuthError, match="opencode auth login"): - _check_auth() - - -# --------------------------------------------------------------------------- -# _check_binary_compat -# --------------------------------------------------------------------------- - - -class TestCheckBinaryCompat: - def test_skips_when_already_checked(self) -> None: - oc_module._compat_checked = True - _check_binary_compat() - - def test_returns_early_when_no_binary(self) -> None: - with patch("shutil.which", return_value=None): - _check_binary_compat() - assert oc_module._compat_checked is True - - def test_v1x_detected_ok(self) -> None: - mock_result = MagicMock(stdout="1.18.14", stderr="") - with patch("shutil.which", return_value="/usr/local/bin/opencode"): - with patch("factory.runners.opencode.subprocess.run", return_value=mock_result): - _check_binary_compat() - assert oc_module._compat_checked is True - - def test_v0x_warns(self) -> None: - mock_result = MagicMock(stdout="opencode version v0.0.55", stderr="") - with patch("shutil.which", return_value="/usr/local/bin/opencode"): - with patch("factory.runners.opencode.subprocess.run", return_value=mock_result): - _check_binary_compat() - assert oc_module._compat_checked is True - - def test_file_not_found_handled(self) -> None: - with patch("shutil.which", return_value="/usr/local/bin/opencode"): - with patch( - "factory.runners.opencode.subprocess.run", - side_effect=FileNotFoundError, - ): - _check_binary_compat() - assert oc_module._compat_checked is True - - def test_timeout_handled(self) -> None: - import subprocess - - with patch("shutil.which", return_value="/usr/local/bin/opencode"): - with patch( - "factory.runners.opencode.subprocess.run", - side_effect=subprocess.TimeoutExpired(cmd="opencode", timeout=10), - ): - _check_binary_compat() - assert oc_module._compat_checked is True - - -# --------------------------------------------------------------------------- -# OpenCodeRunner.metadata -# --------------------------------------------------------------------------- - - -class TestMetadata: - def test_metadata_v1x(self) -> None: - meta = OpenCodeRunner.metadata() - assert meta.name == "opencode" - assert meta.supports_model_override is True - assert meta.supports_session_name is True - assert meta.supports_session_resume is True - assert meta.supports_background is False - assert meta.required_env_vars == [] - assert "opencode.ai/install" in meta.install_hint - assert meta.custom_auth_check is not None - - -# --------------------------------------------------------------------------- -# OpenCodeRunner.build_command -# --------------------------------------------------------------------------- - - -class TestBuildCommand: - def test_basic_command_structure(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - cmd, env, temp_files = runner.build_command( - AgentRunRequest( - prompt="You are the CEO.", - task="Run experiment", - cwd=tmp_path, - role="ceo", - ) - ) - - assert cmd[0] == "opencode" - assert cmd[1] == "run" - assert cmd[2] == "Run experiment" - assert "--format" in cmd - assert "json" in cmd - assert "--dir" in cmd - assert str(tmp_path) in cmd - assert "--auto" in cmd - assert "-p" not in cmd - assert "-c" not in cmd - assert "-q" not in cmd - assert "VIRTUAL_ENV" not in env - - agents_md = tmp_path / "AGENTS.md" - assert agents_md in temp_files - assert agents_md.exists() - assert agents_md.read_text() == "You are the CEO." - - def test_model_override(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - cmd, _, _ = runner.build_command( - AgentRunRequest( - prompt="test", - task="test", - cwd=tmp_path, - role="ceo", - model="anthropic/claude-sonnet-4-20250514", - ) - ) - assert "--model" in cmd - idx = cmd.index("--model") - assert cmd[idx + 1] == "anthropic/claude-sonnet-4-20250514" - - def test_session_name(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - cmd, _, _ = runner.build_command( - AgentRunRequest( - prompt="test", - task="test", - cwd=tmp_path, - role="ceo", - session_name="my-session", - ) - ) - assert "--title" in cmd - idx = cmd.index("--title") - assert cmd[idx + 1] == "my-session" - - def test_session_resume(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - cmd, _, _ = runner.build_command( - AgentRunRequest( - prompt="test", - task="test", - cwd=tmp_path, - role="ceo", - resume_session_id="sess-abc", - ) - ) - assert "--session" in cmd - idx = cmd.index("--session") - assert cmd[idx + 1] == "sess-abc" - - def test_session_continue(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - cmd, _, _ = runner.build_command( - AgentRunRequest( - prompt="test", - task="test", - cwd=tmp_path, - role="ceo", - session_id="any", - ) - ) - assert "--continue" in cmd - - def test_no_auto_without_skip_permissions(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - cmd, _, _ = runner.build_command( - AgentRunRequest( - prompt="test", - task="test", - cwd=tmp_path, - role="ceo", - skip_permissions=False, - ) - ) - assert "--auto" not in cmd - - -# --------------------------------------------------------------------------- -# OpenCodeRunner.build_interactive_command -# --------------------------------------------------------------------------- - - -class TestBuildInteractiveCommand: - def test_interactive_no_run_subcommand(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - cmd, _, temp_files = runner.build_interactive_command( - AgentRunRequest( - prompt="You are a test agent.", - task="Start session", - cwd=tmp_path, - role="ceo", - skip_permissions=False, - ) - ) - assert cmd[0] == "opencode" - assert "run" not in cmd - assert "--format" not in cmd - assert "--auto" not in cmd - assert "--prompt" in cmd - prompt_idx = cmd.index("--prompt") - assert cmd[prompt_idx + 1] == "Start session" - assert "--dir" not in cmd - assert cmd[-1] == str(tmp_path) - - agents_md = tmp_path / "AGENTS.md" - assert agents_md in temp_files - assert agents_md.exists() - assert agents_md.read_text() == "You are a test agent." - - def test_interactive_no_title_flag(self, tmp_path: Path) -> None: - """--title is only valid for 'opencode run', not the base TUI command.""" - runner = OpenCodeRunner() - cmd, _, _ = runner.build_interactive_command( - AgentRunRequest( - prompt="test", - task="test", - cwd=tmp_path, - role="ceo", - session_name="factory: discover run-123", - ) - ) - assert "--title" not in cmd - - def test_interactive_auto_with_skip_permissions(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - cmd, _, _ = runner.build_interactive_command( - AgentRunRequest( - prompt="test", - task="test", - cwd=tmp_path, - role="ceo", - skip_permissions=True, - ) - ) - assert "--auto" in cmd - - def test_interactive_no_auto_without_skip_permissions(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - cmd, _, _ = runner.build_interactive_command( - AgentRunRequest( - prompt="test", - task="test", - cwd=tmp_path, - role="ceo", - skip_permissions=False, - ) - ) - assert "--auto" not in cmd - - def test_interactive_model_override(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - cmd, _, _ = runner.build_interactive_command( - AgentRunRequest( - prompt="test", - task="test", - cwd=tmp_path, - role="ceo", - model="openai/gpt-4o", - ) - ) - assert "--model" in cmd - idx = cmd.index("--model") - assert cmd[idx + 1] == "openai/gpt-4o" - - -# --------------------------------------------------------------------------- -# OpenCodeRunner.headless -# --------------------------------------------------------------------------- - - -class TestOpenCodeHeadless: - async def test_dry_run_returns_stub( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("FACTORY_OPENCODE_DRY_RUN", "1") - (tmp_path / ".factory").mkdir() - runner = OpenCodeRunner(project_path=tmp_path) - result = await runner.headless( - AgentRunRequest( - prompt="Test prompt", - task="Test task", - cwd=tmp_path, - role="researcher", - project_path=tmp_path, - ) - ) - assert result.return_code == 0 - assert "[DRY-RUN]" in result.stdout - assert "researcher" in result.stdout - - async def test_background_returns_error( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("FACTORY_OPENCODE_DRY_RUN", "1") - runner = OpenCodeRunner() - result = await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="builder", - extras={"background": True}, - ) - ) - assert result.return_code == 1 - assert "--bg is not supported" in result.stdout - - async def test_tmux_persist_returns_error( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - runner = OpenCodeRunner() - result = await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="ceo", - extras={"tmux_persist": True}, - ) - ) - assert result.return_code == 1 - assert "--tmux-persist is not supported" in result.stdout - - async def test_headless_calls_run_subprocess( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("ANTHROPIC_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_OPENCODE_DRY_RUN", raising=False) - (tmp_path / ".factory").mkdir() - - runner = OpenCodeRunner(project_path=tmp_path) - with patch("factory.runners.opencode._check_auth"): - with patch( - "factory.runners.opencode.run_subprocess", - new_callable=AsyncMock, - ) as mock_run: - mock_run.return_value = AgentRunResult(stdout="output", return_code=0) - result = await runner.headless( - AgentRunRequest( - prompt="You are a test agent.", - task="Say hello", - cwd=tmp_path, - role="researcher", - timeout=60.0, - project_path=tmp_path, - ) - ) - - assert result.return_code == 0 - assert result.stdout == "output" - - call_kwargs = mock_run.call_args.kwargs - assert call_kwargs["runner_name"] == "opencode" - assert call_kwargs["role"] == "researcher" - assert call_kwargs["timeout"] == 60.0 - assert call_kwargs["sanitize"] is True - cmd = mock_run.call_args[0][0] - assert cmd[0] == "opencode" - assert cmd[1] == "run" - assert "--format" in cmd - assert "-q" not in cmd - - async def test_headless_raises_without_auth( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - for var in oc_module._PROVIDER_ENV_VARS: - monkeypatch.delenv(var, raising=False) - monkeypatch.delenv("FACTORY_OPENCODE_DRY_RUN", raising=False) - - runner = OpenCodeRunner() - with patch("factory.runners.opencode._check_binary_compat"): - with patch("factory.runners.opencode.Path.home", return_value=Path("/nonexistent")): - with pytest.raises(OpenCodeAuthError): - await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="researcher", - ) - ) - - async def test_ceiling_exceeded_returns_error( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.delenv("FACTORY_OPENCODE_DRY_RUN", raising=False) - (tmp_path / ".factory").mkdir() - - from factory.runners.usage import CeilingExceededError - - runner = OpenCodeRunner(project_path=tmp_path) - with patch("factory.runners.opencode._check_auth"): - with patch( - "factory.runners.opencode.check_ceilings", - side_effect=CeilingExceededError( - "per-cycle", - 8, - 8, - "FACTORY_OPENCODE_MAX_INVOCATIONS_PER_CYCLE", - "opencode", - ), - ): - with patch.object(runner, "_emit_ceiling_event"): - result = await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="researcher", - project_path=tmp_path, - ) - ) - assert result.return_code == 1 - assert "ceiling exceeded" in result.stdout - - -# --------------------------------------------------------------------------- -# OpenCodeRunner.interactive_run -# --------------------------------------------------------------------------- - - -class TestOpenCodeInteractive: - def test_dry_run( - self, - tmp_path: Path, - monkeypatch: pytest.MonkeyPatch, - capsys: pytest.CaptureFixture[str], - ) -> None: - monkeypatch.setenv("FACTORY_OPENCODE_DRY_RUN", "1") - runner = OpenCodeRunner() - code = runner.interactive_run( - AgentRunRequest( - prompt="Test prompt", - task="Test task", - cwd=tmp_path, - role="ceo", - ) - ) - assert code == 0 - captured = capsys.readouterr() - assert "[DRY-RUN]" in captured.out - - def test_interactive_run_calls_subprocess( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.delenv("FACTORY_OPENCODE_DRY_RUN", raising=False) - monkeypatch.setenv("ANTHROPIC_API_KEY", "test") - (tmp_path / ".factory").mkdir() - - runner = OpenCodeRunner(project_path=tmp_path) - with patch("factory.runners.opencode._check_auth"): - with patch("factory.runners.opencode.subprocess.run") as mock_run: - mock_run.return_value = MagicMock(returncode=0) - code = runner.interactive_run( - AgentRunRequest( - prompt="You are the CEO.", - task="Start session", - cwd=tmp_path, - role="ceo", - project_path=tmp_path, - ) - ) - assert code == 0 - cmd = mock_run.call_args[0][0] - assert cmd[0] == "opencode" - assert "run" not in cmd - assert "--dir" not in cmd - assert cmd[-1] == str(tmp_path) - assert "-p" not in cmd - assert "-c" not in cmd - - -# --------------------------------------------------------------------------- -# Token guardrails (usage integration) -# --------------------------------------------------------------------------- - - -class TestTokenGuardrails: - async def test_usage_logging_on_headless( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("FACTORY_OPENCODE_DRY_RUN", "1") - factory_dir = tmp_path / ".factory" - factory_dir.mkdir() - runner = OpenCodeRunner(project_path=tmp_path) - - await runner.headless( - AgentRunRequest( - prompt="test", - task="test", - cwd=tmp_path, - role="researcher", - project_path=tmp_path, - ) - ) - - usage_log = factory_dir / "opencode_usage.jsonl" - assert usage_log.exists() - entry = json.loads(usage_log.read_text().strip()) - assert entry["role"] == "researcher" - assert entry["dry_run"] is True - - -# --------------------------------------------------------------------------- -# is_opencode_dry_run -# --------------------------------------------------------------------------- - - -class TestIsOpencodeDryRun: - def test_true(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("FACTORY_OPENCODE_DRY_RUN", "1") - assert is_opencode_dry_run() is True - - def test_false(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.delenv("FACTORY_OPENCODE_DRY_RUN", raising=False) - assert is_opencode_dry_run() is False - - def test_true_word(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("FACTORY_OPENCODE_DRY_RUN", "true") - assert is_opencode_dry_run() is True - - def test_yes(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("FACTORY_OPENCODE_DRY_RUN", "yes") - assert is_opencode_dry_run() is True - - -# --------------------------------------------------------------------------- -# OpenCodeRunner.__init__ (cycle_start resolution) -# --------------------------------------------------------------------------- - - -class TestOpenCodeRunnerInit: - def test_init_with_explicit_cycle_start(self) -> None: - ts = datetime(2026, 1, 1, tzinfo=timezone.utc) - runner = OpenCodeRunner(cycle_start=ts) - assert runner.cycle_start == ts - - def test_init_with_project_path(self, tmp_path: Path) -> None: - with patch("factory.runners.opencode.OpenCodeRunner.__init__.__wrapped__", create=True): - runner = OpenCodeRunner(project_path=tmp_path) - assert runner.cycle_start is not None - - def test_init_default(self) -> None: - runner = OpenCodeRunner() - assert runner.cycle_start is not None diff --git a/tests/test_plugin_agents.py b/tests/test_plugin_agents.py index 7cb4c7508..2cad01fac 100644 --- a/tests/test_plugin_agents.py +++ b/tests/test_plugin_agents.py @@ -5,13 +5,8 @@ from factory.agents.plugin import ( AgentMeta, - _READ_ONLY_ROLES, - _WORKSPACE_WRITE_ROLES, - _sandbox_mode, check_agents_in_sync, - check_codex_agents_in_sync, generate_agent_content, - generate_codex_agent_toml, load_agent_config, ) from factory.agents.runner import AgentRole, _PROMPTS_DIR @@ -185,147 +180,3 @@ def test_rejects_invalid_role(self, tmp_path, monkeypatch): assert rc == 1 -class TestSandboxMode: - def test_read_only_roles(self): - for role in _READ_ONLY_ROLES: - assert _sandbox_mode(role) == "read-only" - - def test_workspace_write_roles(self): - for role in _WORKSPACE_WRITE_ROLES: - assert _sandbox_mode(role) == "workspace-write" - - def test_all_known_roles_covered(self): - config = load_agent_config() - for role in config: - assert role in _READ_ONLY_ROLES or role in _WORKSPACE_WRITE_ROLES, ( - f"{role} is not in _READ_ONLY_ROLES or _WORKSPACE_WRITE_ROLES" - ) - assert not (role in _READ_ONLY_ROLES and role in _WORKSPACE_WRITE_ROLES), ( - f"{role} is in both _READ_ONLY_ROLES and _WORKSPACE_WRITE_ROLES" - ) - - def test_unknown_role_defaults_to_read_only(self): - assert _sandbox_mode("nonexistent_role") == "read-only" - - def test_researcher_is_read_only(self): - assert _sandbox_mode("researcher") == "read-only" - - def test_builder_is_workspace_write(self): - assert _sandbox_mode("builder") == "workspace-write" - - def test_ceo_is_workspace_write(self): - assert _sandbox_mode("ceo") == "workspace-write" - - -class TestGenerateCodexAgentToml: - def test_generates_valid_toml_structure(self): - content = generate_codex_agent_toml("researcher") - assert 'name = "factory-researcher"' in content - assert "sandbox_mode" in content - assert "developer_instructions" in content - - def test_has_generated_comment(self): - content = generate_codex_agent_toml("builder") - assert "GENERATED FILE" in content - assert "factory/agents/prompts/builder.md" in content - - def test_sandbox_mode_matches_role(self): - for role in ALL_ROLES: - content = generate_codex_agent_toml(role) - expected_mode = _sandbox_mode(role) - assert f'sandbox_mode = "{expected_mode}"' in content, ( - f"{role}: expected sandbox_mode={expected_mode}" - ) - - def test_name_prefixed_with_factory(self): - for role in ALL_ROLES: - content = generate_codex_agent_toml(role) - assert f'name = "factory-{role}"' in content - - def test_contains_prompt_heading(self): - for role in ALL_ROLES: - source = (_PROMPTS_DIR / f"{role}.md").read_text() - first_line = source.strip().splitlines()[0] - generated = generate_codex_agent_toml(role) - assert first_line in generated, ( - f"{role}: generated TOML does not include first line of source prompt" - ) - - def test_has_prerequisite_note(self): - content = generate_codex_agent_toml("builder") - assert "uv tool install" in content - - def test_unknown_role_raises(self): - with pytest.raises(ValueError, match="Unknown agent role"): - generate_codex_agent_toml("nonexistent") - - def test_description_present(self): - for role in ALL_ROLES: - content = generate_codex_agent_toml(role) - assert 'description = "' in content - - def test_multiline_instructions(self): - content = generate_codex_agent_toml("ceo") - assert "developer_instructions = '''" in content - assert content.rstrip().endswith("'''") - - -class TestCheckCodexAgentsInSync: - def test_passes_when_all_generated(self, tmp_path): - config = load_agent_config() - for role in config: - (tmp_path / f"{role}.toml").write_text(generate_codex_agent_toml(role)) - assert check_codex_agents_in_sync(tmp_path) == [] - - def test_detects_missing_file(self, tmp_path): - out_of_sync = check_codex_agents_in_sync(tmp_path) - assert len(out_of_sync) == len(load_agent_config()) - - def test_detects_stale_file(self, tmp_path): - config = load_agent_config() - for role in config: - (tmp_path / f"{role}.toml").write_text(generate_codex_agent_toml(role)) - (tmp_path / "builder.toml").write_text("stale content") - out_of_sync = check_codex_agents_in_sync(tmp_path) - assert out_of_sync == ["builder"] - - def test_none_dir_returns_empty(self): - assert check_codex_agents_in_sync(None) == [] - - -class TestCmdInstallCodex: - def test_installs_codex_agents(self, tmp_path, monkeypatch): - monkeypatch.setattr("pathlib.Path.home", lambda: tmp_path) - from argparse import Namespace - - from factory.cli import cmd_install - - rc = cmd_install(Namespace(role=None, runner="codex")) - assert rc == 0 - agents_dir = tmp_path / ".codex" / "agents" - for role in ALL_ROLES: - agent_file = agents_dir / f"factory-{role}.toml" - assert agent_file.exists(), f"Missing TOML agent file for {role}" - content = agent_file.read_text() - assert f'name = "factory-{role}"' in content - - def test_installs_single_codex_role(self, tmp_path, monkeypatch): - monkeypatch.setattr("pathlib.Path.home", lambda: tmp_path) - from argparse import Namespace - - from factory.cli import cmd_install - - rc = cmd_install(Namespace(role="builder", runner="codex")) - assert rc == 0 - agents_dir = tmp_path / ".codex" / "agents" - assert (agents_dir / "factory-builder.toml").exists() - assert not (agents_dir / "factory-ceo.toml").exists() - - def test_rejects_invalid_codex_role(self, tmp_path, monkeypatch): - monkeypatch.setattr("pathlib.Path.home", lambda: tmp_path) - from argparse import Namespace - - from factory.cli import cmd_install - - rc = cmd_install(Namespace(role="nonexistent", runner="codex")) - assert rc == 1 diff --git a/tests/test_runner_e2e.py b/tests/test_runner_e2e.py index e9991680f..dd4205e19 100644 --- a/tests/test_runner_e2e.py +++ b/tests/test_runner_e2e.py @@ -28,7 +28,7 @@ from factory.agents.runner import invoke_agent from factory.runners import get_all_runner_meta, get_available_runners, get_runner -_DRY_RUN_VARS = ["FACTORY_BOB_DRY_RUN", "FACTORY_CODEX_DRY_RUN", "FACTORY_OPENCODE_DRY_RUN"] +_DRY_RUN_VARS = ["FACTORY_CODEX_DRY_RUN", "FACTORY_OPENCODE_DRY_RUN"] @pytest.fixture(autouse=True) @@ -62,47 +62,8 @@ def _runner_has_auth(name: str) -> bool: if not meta.is_available(): return False - if name == "bob": - # Bob stores auth in ~/.bob/, not env vars — if the binary responds, it's authed - try: - result = subprocess.run( - ["bob", "--version"], - capture_output=True, - text=True, - timeout=10, - ) - return result.returncode == 0 - except (FileNotFoundError, subprocess.TimeoutExpired): - return False - - if name == "codex": - # Codex uses ChatGPT OAuth — check via login status - if os.environ.get("CODEX_API_KEY") or os.environ.get("OPENAI_API_KEY"): - return True - try: - result = subprocess.run( - ["codex", "login", "status"], - capture_output=True, - text=True, - timeout=10, - ) - return result.returncode == 0 - except (FileNotFoundError, subprocess.TimeoutExpired): - return False - - if name == "opencode": - if os.environ.get("OPENAI_API_KEY"): - return True - try: - result = subprocess.run( - ["zsh", "-c", "source ~/.zshrc 2>/dev/null && echo $OPENAI_API_KEY"], - capture_output=True, - text=True, - timeout=5, - ) - return bool(result.stdout.strip()) - except (FileNotFoundError, subprocess.TimeoutExpired): - return False + if name in ("bob", "codex", "opencode"): + return False # Claude — if binary is available, auth is handled by the CLI itself return True @@ -317,27 +278,21 @@ def test_runners_list_json() -> None: assert code == 0 data = json.loads(buf.getvalue()) assert isinstance(data, list) - assert len(data) >= 4 + assert len(data) >= 1 names = {r["name"] for r in data} assert "claude" in names - assert "bob" in names - assert "codex" in names - assert "opencode" in names def test_get_available_runners_includes_all_builtins() -> None: - """get_available_runners includes all 4 built-in runners.""" + """get_available_runners includes claude runner.""" runners = get_available_runners() assert "claude" in runners - assert "bob" in runners - assert "codex" in runners - assert "opencode" in runners def test_runner_metadata_consistency() -> None: """All runners have consistent metadata.""" meta_list = get_all_runner_meta() - assert len(meta_list) >= 4 + assert len(meta_list) >= 1 names = set() for m in meta_list: @@ -554,24 +509,8 @@ async def test_headless_produces_output(runner_name: str, sample_project: Path) def _cli_env() -> dict[str, str]: - """Build subprocess env with PATH that includes ~/go/bin for opencode.""" + """Build subprocess env for CLI tests.""" env = os.environ.copy() - go_bin = str(Path.home() / "go" / "bin") - if go_bin not in env.get("PATH", ""): - env["PATH"] = go_bin + ":" + env["PATH"] - if not env.get("OPENAI_API_KEY"): - try: - result = subprocess.run( - ["zsh", "-c", "source ~/.zshrc 2>/dev/null && echo $OPENAI_API_KEY"], - capture_output=True, - text=True, - timeout=5, - ) - key = result.stdout.strip() - if key: - env["OPENAI_API_KEY"] = key - except (FileNotFoundError, subprocess.TimeoutExpired): - pass for var in _DRY_RUN_VARS: env.pop(var, None) return env @@ -582,9 +521,6 @@ def _cli_env() -> dict[str, str]: def test_factory_agent_cli_per_runner(runner_name: str, sample_project: Path) -> None: """factory agent researcher via CLI subprocess for each runner.""" env = _cli_env() - if runner_name == "codex": - env.pop("OPENAI_API_KEY", None) - env.pop("CODEX_API_KEY", None) result = subprocess.run( [ "uv", @@ -640,7 +576,7 @@ def test_factory_eval_runs(sample_project: Path) -> None: def test_factory_runners_list_all_present() -> None: - """factory runners list --json shows all 4 runners with correct metadata.""" + """factory runners list --json shows registered runners with correct metadata.""" result = subprocess.run( ["uv", "run", "factory", "runners", "list", "--json"], capture_output=True, @@ -651,8 +587,7 @@ def test_factory_runners_list_all_present() -> None: data = json.loads(result.stdout) assert isinstance(data, list) names = {r["name"] for r in data} - for expected in ("claude", "bob", "codex", "opencode"): - assert expected in names, f"runner '{expected}' missing from list" + assert "claude" in names, "runner 'claude' missing from list" for runner in data: assert "name" in runner assert "display_name" in runner diff --git a/tests/test_runners.py b/tests/test_runners.py index 8900793a5..24d373a4a 100644 --- a/tests/test_runners.py +++ b/tests/test_runners.py @@ -9,16 +9,7 @@ import pytest from factory.models import AgentRunRequest, AgentRunResult -from factory.runners import ClaudeRunner, BobRunner, get_runner, is_dry_run -from factory.runners.opencode import OpenCodeRunner -from factory.runners.usage import ( - CeilingExceededError, - CeilingWarning, - check_ceilings, - count_cycle_invocations, - get_usage_log_path, - log_usage, -) +from factory.runners import ClaudeRunner, get_runner class TestGetRunner: @@ -30,17 +21,13 @@ def test_explicit_claude(self) -> None: runner = get_runner("claude") assert runner.name == "claude" - def test_explicit_bob(self) -> None: - runner = get_runner("bob") - assert runner.name == "bob" - def test_from_env_var(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("FACTORY_RUNNER", "bob") + monkeypatch.setenv("FACTORY_RUNNER", "claude") runner = get_runner() - assert runner.name == "bob" + assert runner.name == "claude" def test_explicit_overrides_env(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("FACTORY_RUNNER", "bob") + monkeypatch.setenv("FACTORY_RUNNER", "claude") runner = get_runner("claude") assert runner.name == "claude" @@ -246,581 +233,6 @@ async def test_headless_subprocess_env_suppresses_telemetry(self, tmp_path: Path assert call_kwargs["env"]["TELEMETRY_PLATFORM"] == "" -class TestBobRunner: - def test_is_dry_run_true(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("FACTORY_BOB_DRY_RUN", "1") - assert is_dry_run() is True - - def test_is_dry_run_false(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.delenv("FACTORY_BOB_DRY_RUN", raising=False) - assert is_dry_run() is False - - def test_interactive_run_dry_run( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] - ) -> None: - """interactive_run prints dry-run message and returns 0.""" - monkeypatch.setenv("FACTORY_BOB_DRY_RUN", "1") - (tmp_path / ".factory").mkdir() - - runner = BobRunner() - - code = runner.interactive_run( - AgentRunRequest( - prompt="Test prompt", - task="Test task", - cwd=tmp_path, - role="ceo", - ) - ) - - assert code == 0 - captured = capsys.readouterr() - assert "[DRY-RUN]" in captured.out - - async def test_headless_timeout(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - """BobRunner.headless() handles timeout gracefully.""" - monkeypatch.setenv("BOBSHELL_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_BOB_DRY_RUN", raising=False) - - import factory.runners.bob as bob_module - - bob_module._auth_checked = False - - (tmp_path / ".factory").mkdir() - - # Mock run_subprocess to return an inactivity timeout result - with patch("factory.runners.bob.run_subprocess", new_callable=AsyncMock) as mock_run: - mock_run.return_value = AgentRunResult( - stdout="Agent killed after 0.1s of inactivity", - return_code=1, - ) - - runner = BobRunner() - result = await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="researcher", - timeout=0.1, - ) - ) - - assert result.return_code == 1 - assert "inactivity" in result.stdout.lower() - assert result.usage is None - bob_module._auth_checked = False - - def test_count_cycle_invocations_with_datetime(self, tmp_path: Path) -> None: - """count_cycle_invocations filters by cycle_start datetime.""" - from datetime import datetime, timezone, timedelta - from factory.runners.usage import count_cycle_invocations, get_usage_log_path - import json - - (tmp_path / ".factory").mkdir() - - now = datetime.now(timezone.utc) - old_time = now - timedelta(hours=2) - - log_path = get_usage_log_path(tmp_path) - entries = [ - { - "timestamp": old_time.isoformat(), - "role": "a", - "cwd": str(tmp_path), - "duration_seconds": 1.0, - "exit_code": 0, - "dry_run": False, - }, - { - "timestamp": now.isoformat(), - "role": "b", - "cwd": str(tmp_path), - "duration_seconds": 1.0, - "exit_code": 0, - "dry_run": False, - }, - { - "timestamp": now.isoformat(), - "role": "c", - "cwd": str(tmp_path), - "duration_seconds": 1.0, - "exit_code": 0, - "dry_run": True, - }, - ] - - with open(log_path, "w") as f: - for entry in entries: - f.write(json.dumps(entry) + "\n") - - cycle_start = now - timedelta(hours=1) - count = count_cycle_invocations(tmp_path, cycle_start) - assert count == 1 - - async def test_headless_ceiling_exceeded( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """BobRunner returns error when ceiling exceeded.""" - from datetime import datetime, timezone, timedelta - - monkeypatch.setenv("BOBSHELL_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_BOB_DRY_RUN", raising=False) - monkeypatch.setenv("FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE", "1") - - import factory.runners.bob as bob_module - - bob_module._auth_checked = False - - (tmp_path / ".factory").mkdir() - - # Create runner FIRST with a cycle_start in the past - cycle_start = datetime.now(timezone.utc) - timedelta(seconds=5) - runner = BobRunner(cycle_start=cycle_start) - - # Log entry AFTER cycle_start so it counts - log_usage(tmp_path, "a", tmp_path, 1.0, 0, dry_run=False) - - result = await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="researcher", - ) - ) - - assert result.return_code == 1 - assert "ceiling" in result.stdout.lower() or "exceeded" in result.stdout.lower() - assert result.usage is None - bob_module._auth_checked = False - - async def test_dry_run_returns_stub( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("FACTORY_BOB_DRY_RUN", "1") - - # Create .factory directory for usage log - (tmp_path / ".factory").mkdir() - - runner = BobRunner() - result = await runner.headless( - AgentRunRequest( - prompt="You are a test agent.", - task="Say hello", - cwd=tmp_path, - role="researcher", - ) - ) - - assert result.return_code == 0 - assert "[DRY-RUN]" in result.stdout - assert "researcher" in result.stdout - assert result.usage is None - - async def test_dry_run_logs_usage( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.setenv("FACTORY_BOB_DRY_RUN", "1") - - # Create .factory directory - (tmp_path / ".factory").mkdir() - - runner = BobRunner() - await runner.headless( - AgentRunRequest( - prompt="Test prompt", - task="Test task", - cwd=tmp_path, - role="builder", - ) - ) - - log_path = get_usage_log_path(tmp_path) - assert log_path.exists() - - with open(log_path) as f: - entry = json.loads(f.readline()) - - assert entry["role"] == "builder" - assert entry["dry_run"] is True - assert entry["exit_code"] == 0 - - -class TestUsageTracking: - def test_log_usage_creates_file(self, tmp_path: Path) -> None: - (tmp_path / ".factory").mkdir() - - log_usage(tmp_path, "researcher", tmp_path, 1.5, 0, dry_run=False) - - log_path = get_usage_log_path(tmp_path) - assert log_path.exists() - - with open(log_path) as f: - entry = json.loads(f.readline()) - - assert entry["role"] == "researcher" - assert entry["duration_seconds"] == 1.5 - assert entry["exit_code"] == 0 - assert entry["dry_run"] is False - - def test_count_cycle_invocations_with_start(self, tmp_path: Path) -> None: - from datetime import datetime, timezone, timedelta - - (tmp_path / ".factory").mkdir() - - # Log some entries - log_usage(tmp_path, "a", tmp_path, 1.0, 0, dry_run=False) - log_usage(tmp_path, "b", tmp_path, 1.0, 0, dry_run=False) - log_usage(tmp_path, "c", tmp_path, 1.0, 0, dry_run=True) # dry-run, shouldn't count - - # Count from beginning of the current second - cycle_start = datetime.now(timezone.utc) - timedelta(seconds=5) - count = count_cycle_invocations(tmp_path, cycle_start) - assert count == 2 # dry-run excluded - - def test_count_cycle_invocations_none_returns_zero(self, tmp_path: Path) -> None: - (tmp_path / ".factory").mkdir() - - log_usage(tmp_path, "a", tmp_path, 1.0, 0, dry_run=False) - log_usage(tmp_path, "b", tmp_path, 1.0, 0, dry_run=False) - - # Without cycle_start, returns 0 - count = count_cycle_invocations(tmp_path, None) - assert count == 0 - - -class TestCeilings: - def test_check_ceilings_passes_when_under( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - from datetime import datetime, timezone, timedelta - - (tmp_path / ".factory").mkdir() - monkeypatch.setenv("FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE", "5") - - # Log a few entries (under ceiling) - log_usage(tmp_path, "a", tmp_path, 1.0, 0, dry_run=False) - log_usage(tmp_path, "b", tmp_path, 1.0, 0, dry_run=False) - - # Should not raise - cycle_start = datetime.now(timezone.utc) - timedelta(seconds=5) - check_ceilings(tmp_path, cycle_start) - - def test_check_ceilings_fails_on_cycle( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - from datetime import datetime, timezone, timedelta - - (tmp_path / ".factory").mkdir() - monkeypatch.setenv("FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE", "1") - - log_usage(tmp_path, "a", tmp_path, 1.0, 0, dry_run=False) - - cycle_start = datetime.now(timezone.utc) - timedelta(seconds=5) - with pytest.raises(CeilingExceededError) as exc_info: - check_ceilings(tmp_path, cycle_start) - - assert exc_info.value.ceiling_name == "per-cycle" - assert exc_info.value.env_var == "FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE" - - def test_ceiling_error_message_is_actionable(self) -> None: - error = CeilingExceededError("per-cycle", 5, 5, "FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE") - msg = str(error) - - assert "ceiling exceeded" in msg.lower() - assert "5/5" in msg - assert "FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE=10" in msg # suggests bumping - - -class TestCeilingWarning: - def test_warning_returned_when_cycle_ceiling_near( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """check_ceilings returns CeilingWarning when ≤2 cycle invocations remain.""" - from datetime import datetime, timezone, timedelta - - (tmp_path / ".factory").mkdir() - monkeypatch.setenv("FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE", "4") - - log_usage(tmp_path, "a", tmp_path, 1.0, 0, dry_run=False) - log_usage(tmp_path, "b", tmp_path, 1.0, 0, dry_run=False) - - cycle_start = datetime.now(timezone.utc) - timedelta(seconds=5) - warning = check_ceilings(tmp_path, cycle_start) - - assert warning is not None - assert isinstance(warning, CeilingWarning) - assert warning.ceiling_name == "per-cycle" - assert warning.remaining == 2 - assert warning.limit == 4 - - def test_no_warning_when_sufficient_invocations_remain( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """check_ceilings returns None when >2 invocations remain.""" - from datetime import datetime, timezone, timedelta - - (tmp_path / ".factory").mkdir() - monkeypatch.setenv("FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE", "10") - - log_usage(tmp_path, "a", tmp_path, 1.0, 0, dry_run=False) - - cycle_start = datetime.now(timezone.utc) - timedelta(seconds=5) - warning = check_ceilings(tmp_path, cycle_start) - - assert warning is None - - def test_warning_at_exactly_one_remaining( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """check_ceilings returns CeilingWarning when exactly 1 invocation remains.""" - from datetime import datetime, timezone, timedelta - - (tmp_path / ".factory").mkdir() - monkeypatch.setenv("FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE", "3") - - log_usage(tmp_path, "a", tmp_path, 1.0, 0, dry_run=False) - log_usage(tmp_path, "b", tmp_path, 1.0, 0, dry_run=False) - - cycle_start = datetime.now(timezone.utc) - timedelta(seconds=5) - warning = check_ceilings(tmp_path, cycle_start) - - assert warning is not None - assert warning.ceiling_name == "per-cycle" - assert warning.remaining == 1 - - -class TestBobAuthPreflight: - async def test_auth_check_fails_without_key( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.delenv("FACTORY_BOB_DRY_RUN", raising=False) - monkeypatch.delenv("BOBSHELL_API_KEY", raising=False) - - # Reset the auth check state - import factory.runners.bob as bob_module - - bob_module._auth_checked = False - - # Redirect home so native auth at ~/.bob/settings.json isn't found - monkeypatch.setattr(Path, "home", staticmethod(lambda: tmp_path)) - - (tmp_path / ".factory").mkdir() - - runner = BobRunner() - - from factory.runners.bob import BobAuthError - - with pytest.raises(BobAuthError): - await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="researcher", - ) - ) - - async def test_auth_check_passes_with_key( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.delenv("FACTORY_BOB_DRY_RUN", raising=False) - monkeypatch.setenv("BOBSHELL_API_KEY", "test-key") - - # Reset the auth check state - import factory.runners.bob as bob_module - - bob_module._auth_checked = False - - (tmp_path / ".factory").mkdir() - - # Mock run_subprocess to avoid actual bob invocation - with patch("factory.runners.bob.run_subprocess", new_callable=AsyncMock) as mock_run: - mock_run.return_value = AgentRunResult( - stdout="output", - return_code=0, - ) - - runner = BobRunner() - result = await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="researcher", - ) - ) - - assert result.return_code == 0 - assert result.usage is None - - -class TestKeyPersistence: - """Tests for file-based API key persistence.""" - - def test_persist_key_creates_file( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """Verify _persist_key writes the key to .factory/.bob_auth.""" - monkeypatch.setenv("BOBSHELL_API_KEY", "test-secret-key") - - (tmp_path / ".factory").mkdir() - - from factory.runners.bob import _persist_key - - _persist_key(tmp_path) - - auth_file = tmp_path / ".factory" / ".bob_auth" - assert auth_file.exists() - assert auth_file.read_text() == "test-secret-key" - - # Verify file permissions (chmod 600) - mode = auth_file.stat().st_mode - assert mode & 0o777 == 0o600 - - def test_persist_key_no_op_without_env_var( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """Verify _persist_key does nothing if BOBSHELL_API_KEY is not set.""" - monkeypatch.delenv("BOBSHELL_API_KEY", raising=False) - - (tmp_path / ".factory").mkdir() - - from factory.runners.bob import _persist_key - - _persist_key(tmp_path) - - auth_file = tmp_path / ".factory" / ".bob_auth" - assert not auth_file.exists() - - def test_check_auth_reads_from_file( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """Verify _check_auth falls back to reading from file when env var missing.""" - monkeypatch.delenv("BOBSHELL_API_KEY", raising=False) - - import factory.runners.bob as bob_module - - bob_module._auth_checked = False - - # Create the auth file - (tmp_path / ".factory").mkdir() - auth_file = tmp_path / ".factory" / ".bob_auth" - auth_file.write_text("file-based-key") - - # Change to tmp_path so _find_auth_file can find it - monkeypatch.chdir(tmp_path) - - from factory.runners.bob import _check_auth - - _check_auth() - - # Verify the key was injected into os.environ - assert os.environ.get("BOBSHELL_API_KEY") == "file-based-key" - # Clean up injected env var - monkeypatch.delenv("BOBSHELL_API_KEY", raising=False) - bob_module._auth_checked = False - - def test_check_auth_prefers_env_var( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """Verify env var takes precedence over file.""" - monkeypatch.setenv("BOBSHELL_API_KEY", "env-key") - - import factory.runners.bob as bob_module - - bob_module._auth_checked = False - - # Create the auth file with a different key - (tmp_path / ".factory").mkdir() - auth_file = tmp_path / ".factory" / ".bob_auth" - auth_file.write_text("file-key") - - monkeypatch.chdir(tmp_path) - - from factory.runners.bob import _check_auth - - _check_auth() - - # Env var should still be the original value - assert os.environ.get("BOBSHELL_API_KEY") == "env-key" - bob_module._auth_checked = False - - def test_preflight_error_unchanged_when_no_key( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """Verify BobAuthError is raised when key is missing from both env and file.""" - monkeypatch.delenv("BOBSHELL_API_KEY", raising=False) - - import factory.runners.bob as bob_module - - bob_module._auth_checked = False - - # Redirect home so native auth at ~/.bob/settings.json isn't found - monkeypatch.setattr(Path, "home", staticmethod(lambda: tmp_path)) - - # No .factory directory, no auth file - monkeypatch.chdir(tmp_path) - - from factory.runners.bob import _check_auth, BobAuthError - - with pytest.raises(BobAuthError) as exc_info: - _check_auth() - - assert "BOBSHELL_API_KEY environment variable is not set" in str(exc_info.value) - bob_module._auth_checked = False - - async def test_headless_passes_key_to_subprocess( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """Verify the subprocess env dict contains BOBSHELL_API_KEY from file.""" - monkeypatch.delenv("BOBSHELL_API_KEY", raising=False) - monkeypatch.delenv("FACTORY_BOB_DRY_RUN", raising=False) - - import factory.runners.bob as bob_module - - bob_module._auth_checked = False - - # Create the auth file - (tmp_path / ".factory").mkdir() - auth_file = tmp_path / ".factory" / ".bob_auth" - auth_file.write_text("subprocess-test-key") - - monkeypatch.chdir(tmp_path) - - with patch( - "factory.runners._subprocess.stream_subprocess", new_callable=AsyncMock - ) as mock_stream: - mock_stream.return_value = (b"output", b"") - - with patch( - "factory.runners._subprocess.asyncio.create_subprocess_exec", new_callable=AsyncMock - ) as mock_exec: - mock_proc = AsyncMock() - mock_proc.returncode = 0 - mock_exec.return_value = mock_proc - - runner = BobRunner() - result = await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="researcher", - ) - ) - - # Verify the subprocess was called with env containing the key - call_kwargs = mock_exec.call_args.kwargs - assert "env" in call_kwargs - assert call_kwargs["env"].get("BOBSHELL_API_KEY") == "subprocess-test-key" - assert result.usage is None - - monkeypatch.delenv("BOBSHELL_API_KEY", raising=False) - bob_module._auth_checked = False - - class TestStreamingOutput: """Tests for streaming subprocess output to terminal.""" @@ -1002,57 +414,6 @@ async def test_claude_runner_uses_streaming( assert call_kwargs["stream"] is True assert call_kwargs["prefix"] == "[claude:researcher]" - async def test_bob_runner_uses_streaming( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """BobRunner.headless() streams output when should_stream() is True.""" - monkeypatch.setenv("FACTORY_BOB_DRY_RUN", "1") - monkeypatch.delenv("FACTORY_RUNNER_QUIET", raising=False) - - (tmp_path / ".factory").mkdir() - - runner = BobRunner() - - # For dry-run, streaming doesn't apply — test the non-dry-run path - monkeypatch.delenv("FACTORY_BOB_DRY_RUN", raising=False) - monkeypatch.setenv("BOBSHELL_API_KEY", "test-key") - - import factory.runners.bob as bob_module - - bob_module._auth_checked = False - - with patch("factory.runners._subprocess.should_stream", return_value=True): - with patch( - "factory.runners._subprocess.stream_subprocess", new_callable=AsyncMock - ) as mock_stream: - mock_stream.return_value = (b"output\n", b"") - - with patch( - "factory.runners._subprocess.asyncio.create_subprocess_exec", - new_callable=AsyncMock, - ) as mock_exec: - mock_proc = AsyncMock() - mock_proc.returncode = 0 - mock_exec.return_value = mock_proc - - result = await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="builder", - ) - ) - - # Verify stream_subprocess was called with streaming enabled - mock_stream.assert_called_once() - call_kwargs = mock_stream.call_args.kwargs - assert call_kwargs["stream"] is True - assert call_kwargs["prefix"] == "[bob:builder]" - assert result.usage is None - - bob_module._auth_checked = False - async def test_quiet_mode_disables_streaming( self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch ) -> None: @@ -1321,55 +682,19 @@ async def readline(self) -> bytes: class MockProc: def __init__(self) -> None: self.stdout = MockReader([b"out\n"]) - self.stderr = MockReader([b"err\n"]) - - async def wait(self) -> int: - return 0 - - proc = MockProc() - - with patch("factory.runners._stream.tee_stream", new_callable=AsyncMock) as mock_tee: - await stream_subprocess(proc, stream=False, sanitize=True) # type: ignore[arg-type] - - assert mock_tee.call_count == 2 - for call in mock_tee.call_args_list: - assert call.kwargs["sanitize"] is True - - async def test_bob_runner_passes_sanitize_true( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """BobRunner.headless() passes sanitize=True to run_subprocess.""" - monkeypatch.delenv("FACTORY_BOB_DRY_RUN", raising=False) - monkeypatch.delenv("FACTORY_RUNNER_QUIET", raising=False) - monkeypatch.setenv("BOBSHELL_API_KEY", "test-key") - - (tmp_path / ".factory").mkdir() - - import factory.runners.bob as bob_module - - bob_module._auth_checked = False - - runner = BobRunner() - - with patch("factory.runners.bob.run_subprocess", new_callable=AsyncMock) as mock_run: - mock_run.return_value = AgentRunResult( - stdout="output\n", - return_code=0, - ) - - await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="builder", - ) - ) + self.stderr = MockReader([b"err\n"]) - mock_run.assert_called_once() - assert mock_run.call_args.kwargs["sanitize"] is True + async def wait(self) -> int: + return 0 + + proc = MockProc() + + with patch("factory.runners._stream.tee_stream", new_callable=AsyncMock) as mock_tee: + await stream_subprocess(proc, stream=False, sanitize=True) # type: ignore[arg-type] - bob_module._auth_checked = False + assert mock_tee.call_count == 2 + for call in mock_tee.call_args_list: + assert call.kwargs["sanitize"] is True async def test_claude_runner_sanitizes( self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch @@ -1463,362 +788,6 @@ async def test_max_timeout_backstop(self) -> None: assert "max wall-clock timeout" in result.stdout.lower() -class TestCeilingAccumulationAcrossInvocations: - """Tests that per-cycle ceiling accumulates across invoke_agent calls.""" - - async def test_ceiling_accumulates_across_invoke_agent_calls( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """Verify that invocation counts accumulate across multiple invoke_agent calls. - - This test reproduces the bug from PR #136: each get_runner() call created - a fresh BobRunner with cycle_start=now(), so the ceiling never accumulated. - - With the fix, get_runner() passes project_path to BobRunner, which reads - started_at from .factory/state/cycle.json, ensuring all invocations within - a cycle share the same cycle_start and accumulate correctly. - """ - from unittest.mock import AsyncMock, patch - - from factory.agents.runner import invoke_agent - from factory.ceo_completion import write_cycle_state, create_cycle_state - - monkeypatch.setenv("FACTORY_RUNNER", "bob") - monkeypatch.setenv("BOBSHELL_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_BOB_DRY_RUN", raising=False) - monkeypatch.setenv("FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE", "2") - - # Reset auth check state - import factory.runners.bob as bob_module - - bob_module._auth_checked = False - - # Create project structure - (tmp_path / ".factory").mkdir() - (tmp_path / ".factory" / "state").mkdir() - - # Create a cycle state (simulates an in-flight cycle) - cycle_state = create_cycle_state("improve", "test task", "bob") - write_cycle_state(tmp_path, cycle_state) - - # Create a minimal agent prompt - prompts_dir = tmp_path / ".factory" / "agents" - prompts_dir.mkdir() - (prompts_dir / "researcher.md").write_text("You are a researcher.") - - # Mock run_subprocess to avoid actually calling bob - with patch("factory.runners.bob.run_subprocess", new_callable=AsyncMock) as mock_run: - mock_run.return_value = AgentRunResult( - stdout="output", - return_code=0, - ) - - # First invocation — should succeed (1/2) - stdout1, code1 = await invoke_agent( - "researcher", - "First task", - tmp_path, - runner_name="bob", - ) - assert code1 == 0, f"First invocation failed: {stdout1}" - - # Second invocation — should succeed (2/2) - stdout2, code2 = await invoke_agent( - "researcher", - "Second task", - tmp_path, - runner_name="bob", - ) - assert code2 == 0, f"Second invocation failed: {stdout2}" - - # Third invocation — should fail (3/2 = ceiling exceeded) - stdout3, code3 = await invoke_agent( - "researcher", - "Third task", - tmp_path, - runner_name="bob", - ) - assert code3 == 1, "Third invocation should have hit the ceiling" - assert "ceiling" in stdout3.lower() or "exceeded" in stdout3.lower() - - bob_module._auth_checked = False - - async def test_bobrunner_reads_cycle_start_from_cycle_json( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """Verify BobRunner reads started_at from cycle.json when project_path is provided.""" - - from factory.ceo_completion import write_cycle_state, create_cycle_state - from factory.runners import get_runner - - monkeypatch.setenv("FACTORY_BOB_DRY_RUN", "1") - - # Create project structure - (tmp_path / ".factory").mkdir() - (tmp_path / ".factory" / "state").mkdir() - - # Create a cycle state with a known started_at - cycle_state = create_cycle_state("improve", "test task", "bob") - write_cycle_state(tmp_path, cycle_state) - - # Get runner with project_path - runner = get_runner("bob", project_path=tmp_path) - - # Runner's cycle_start should match the persisted state's started_at - # (allowing for small time differences in serialization) - time_diff = abs((runner.cycle_start - cycle_state.started_at).total_seconds()) - assert time_diff < 1.0, ( - f"cycle_start mismatch: {runner.cycle_start} vs {cycle_state.started_at}" - ) - - async def test_bobrunner_falls_back_to_now_without_cycle_json( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """Verify BobRunner falls back to now() when no cycle.json exists.""" - from datetime import datetime, timezone - - from factory.runners import get_runner - - monkeypatch.setenv("FACTORY_BOB_DRY_RUN", "1") - - # Create project structure but NO cycle.json - (tmp_path / ".factory").mkdir() - - now_before = datetime.now(timezone.utc) - - # Get runner with project_path (but no cycle.json exists) - runner = get_runner("bob", project_path=tmp_path) - - now_after = datetime.now(timezone.utc) - - # Runner's cycle_start should be between now_before and now_after - assert now_before <= runner.cycle_start <= now_after - - -class TestRunnerBgWarnings: - """Tests for background warning messages from non-claude runners.""" - - async def test_opencode_bg_error(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - """OpenCodeRunner returns error when extras['background']=True.""" - runner = OpenCodeRunner() - result = await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="researcher", - extras={"background": True}, - ) - ) - assert result.return_code == 1 - assert "--bg is not supported" in result.stdout - - async def test_bob_bg_warning(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - """BobRunner logs a warning when extras['background']=True.""" - monkeypatch.setenv("FACTORY_BOB_DRY_RUN", "1") - (tmp_path / ".factory").mkdir() - - runner = BobRunner() - with patch("factory.runners.bob.log") as mock_log: - await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="researcher", - extras={"background": True}, - ) - ) - mock_log.warning.assert_any_call( - "bob_bg_not_supported", hint="--bg is a claude-only feature" - ) - - async def test_codex_bg_warning(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - """CodexRunner logs a warning when extras['background']=True.""" - monkeypatch.setenv("FACTORY_CODEX_DRY_RUN", "1") - - from factory.runners.codex import CodexRunner - - runner = CodexRunner() - with patch("factory.runners.codex.log") as mock_log: - await runner.headless( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - role="researcher", - extras={"background": True}, - ) - ) - mock_log.warning.assert_any_call( - "codex_bg_not_supported", hint="--bg is a claude-only feature" - ) - - -class TestOpenCodeInteractive: - """Tests for OpenCodeRunner.interactive_run() — prompt delivery.""" - - def test_interactive_run_passes_prompt( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """interactive_run() writes prompt to AGENTS.md and passes task via --prompt.""" - monkeypatch.setenv("OPENAI_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_OPENCODE_DRY_RUN", raising=False) - runner = OpenCodeRunner() - - with patch("factory.runners.opencode.subprocess.run") as mock_run: - mock_run.return_value = type("Result", (), {"returncode": 0})() - code = runner.interactive_run( - AgentRunRequest( - prompt="You are the CEO.", - task="Start session", - cwd=tmp_path, - ) - ) - - assert code == 0 - cmd = mock_run.call_args[0][0] - assert cmd[0] == "opencode" - assert "--prompt" in cmd - prompt_idx = cmd.index("--prompt") - assert cmd[prompt_idx + 1] == "Start session" - assert not (tmp_path / "AGENTS.md").exists() - - def test_interactive_run_passes_cwd( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """interactive_run() passes --dir with the cwd.""" - monkeypatch.setenv("OPENAI_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_OPENCODE_DRY_RUN", raising=False) - runner = OpenCodeRunner() - - with patch("factory.runners.opencode.subprocess.run") as mock_run: - mock_run.return_value = type("Result", (), {"returncode": 0})() - runner.interactive_run( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - ) - ) - - cmd = mock_run.call_args[0][0] - assert "--dir" not in cmd - assert cmd[-1] == str(tmp_path) - - def test_interactive_run_dry_run( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch, capsys: pytest.CaptureFixture[str] - ) -> None: - """interactive_run() prints dry-run message and returns 0.""" - monkeypatch.setenv("FACTORY_OPENCODE_DRY_RUN", "1") - runner = OpenCodeRunner() - - code = runner.interactive_run( - AgentRunRequest( - prompt="Test prompt", - task="Test task", - cwd=tmp_path, - ) - ) - - assert code == 0 - captured = capsys.readouterr() - assert "[DRY-RUN]" in captured.out - - -class TestBobInteractivePrompt: - """Tests for BobRunner.interactive_run() — prompt delivery.""" - - def test_interactive_run_passes_prompt_via_i_flag( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - """interactive_run() passes the prompt via -i flag.""" - monkeypatch.setenv("BOBSHELL_API_KEY", "test-key") - monkeypatch.delenv("FACTORY_BOB_DRY_RUN", raising=False) - - import factory.runners.bob as bob_module - - bob_module._auth_checked = False - - (tmp_path / ".factory").mkdir() - runner = BobRunner() - - with patch("subprocess.run") as mock_run: - mock_run.return_value = type("Result", (), {"returncode": 0})() - code = runner.interactive_run( - AgentRunRequest( - prompt="You are the CEO.", - task="Start session", - cwd=tmp_path, - ) - ) - - assert code == 0 - cmd = mock_run.call_args[0][0] - assert cmd[0] == "bob" - assert "-i" in cmd - i_idx = cmd.index("-i") - full_prompt = cmd[i_idx + 1] - assert "You are the CEO." in full_prompt - assert "Start session" in full_prompt - - bob_module._auth_checked = False - - -class TestBobMetaAuthCheck: - """Tests for BobRunner.metadata().check_auth() — file-based auth support.""" - - def test_check_auth_true_with_env_var(self, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("BOBSHELL_API_KEY", "test-key") - meta = BobRunner.metadata() - assert meta.check_auth() is True - - def test_check_auth_true_with_bob_config( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.delenv("BOBSHELL_API_KEY", raising=False) - bob_dir = tmp_path / ".bob" - bob_dir.mkdir() - (bob_dir / "settings.json").write_text("{}") - monkeypatch.setattr(Path, "home", staticmethod(lambda: tmp_path)) - - meta = BobRunner.metadata() - assert meta.check_auth() is True - - def test_check_auth_true_with_factory_auth_file( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.delenv("BOBSHELL_API_KEY", raising=False) - monkeypatch.chdir(tmp_path) - (tmp_path / ".factory").mkdir() - (tmp_path / ".factory" / ".bob_auth").write_text("file-key") - - meta = BobRunner.metadata() - assert meta.check_auth() is True - - def test_check_auth_false_when_nothing_configured( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.delenv("BOBSHELL_API_KEY", raising=False) - monkeypatch.chdir(tmp_path) - monkeypatch.setattr(Path, "home", staticmethod(lambda: tmp_path)) - - meta = BobRunner.metadata() - assert meta.check_auth() is False - - def test_check_auth_false_with_empty_auth_file( - self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch - ) -> None: - monkeypatch.delenv("BOBSHELL_API_KEY", raising=False) - monkeypatch.chdir(tmp_path) - monkeypatch.setattr(Path, "home", staticmethod(lambda: tmp_path)) - (tmp_path / ".factory").mkdir() - (tmp_path / ".factory" / ".bob_auth").write_text(" \n ") - - meta = BobRunner.metadata() - assert meta.check_auth() is False - - class TestRunnerMetaCustomAuthCheck: """Tests for RunnerMeta.custom_auth_check support.""" @@ -2284,146 +1253,6 @@ async def test_tmux_command_includes_disallowed_tools(self, tmp_path: Path) -> N assert "Agent" in wrapper_content -class TestBobBuildInteractiveCommand: - """Tests for BobRunner.build_interactive_command().""" - - def test_base_command_structure(self, tmp_path: Path) -> None: - runner = BobRunner() - cmd, _, _ = runner.build_interactive_command( - AgentRunRequest( - prompt="You are the CEO.", - task="Start session", - cwd=tmp_path, - ) - ) - - assert cmd[0] == "bob" - assert "--chat-mode=code" in cmd - assert "-i" in cmd - i_idx = cmd.index("-i") - full_prompt = cmd[i_idx + 1] - assert "You are the CEO." in full_prompt - assert "Start session" in full_prompt - assert "## Current Task" in full_prompt - - def test_yolo_flag(self, tmp_path: Path) -> None: - runner = BobRunner() - cmd, _, _ = runner.build_interactive_command( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - skip_permissions=True, - ) - ) - - assert "--yolo" in cmd - - def test_no_yolo_without_skip(self, tmp_path: Path) -> None: - runner = BobRunner() - cmd, _, _ = runner.build_interactive_command( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - skip_permissions=False, - ) - ) - - assert "--yolo" not in cmd - - def test_env_uses_dict(self, tmp_path: Path) -> None: - runner = BobRunner() - _, env, _ = runner.build_interactive_command( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - ) - ) - - assert isinstance(env, dict) - assert "PATH" in env - - def test_uses_i_flag_not_p(self, tmp_path: Path) -> None: - runner = BobRunner() - cmd, _, _ = runner.build_interactive_command( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - ) - ) - - assert "-i" in cmd - assert "-p" not in cmd - - -class TestOpenCodeBuildInteractiveCommand: - """Tests for OpenCodeRunner.build_interactive_command().""" - - def test_base_command_structure(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - cmd, _, temp_files = runner.build_interactive_command( - AgentRunRequest( - prompt="You are the CEO.", - task="Start session", - cwd=tmp_path, - ) - ) - - assert cmd[0] == "opencode" - assert "--prompt" in cmd - prompt_idx = cmd.index("--prompt") - assert cmd[prompt_idx + 1] == "Start session" - assert "--dir" not in cmd - assert cmd[-1] == str(tmp_path) - assert "-q" not in cmd - - agents_md = tmp_path / "AGENTS.md" - assert agents_md in temp_files - assert agents_md.exists() - assert agents_md.read_text() == "You are the CEO." - - def test_env_strips_virtual_env(self, tmp_path: Path, monkeypatch: pytest.MonkeyPatch) -> None: - monkeypatch.setenv("VIRTUAL_ENV", "/some/venv") - runner = OpenCodeRunner() - _, env, _ = runner.build_interactive_command( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - ) - ) - - assert "VIRTUAL_ENV" not in env - - def test_no_quiet_flag(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - cmd, _, _ = runner.build_interactive_command( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - ) - ) - - assert "-q" not in cmd - - def test_temp_files_contains_agents_md(self, tmp_path: Path) -> None: - runner = OpenCodeRunner() - _, _, temp_files = runner.build_interactive_command( - AgentRunRequest( - prompt="Test", - task="Test", - cwd=tmp_path, - ) - ) - - assert len(temp_files) == 1 - assert temp_files[0] == tmp_path / "AGENTS.md" - - class TestGetRunnerChoices: """Tests for get_runner_choices() — returns sorted list of runner names.""" @@ -2438,11 +1267,7 @@ def test_returns_sorted_list(self, monkeypatch: pytest.MonkeyPatch) -> None: choices = get_runner_choices() assert isinstance(choices, list) assert choices == sorted(choices) - # All built-in runners should be present assert "claude" in choices - assert "bob" in choices - assert "codex" in choices - assert "opencode" in choices def test_returns_strings(self, monkeypatch: pytest.MonkeyPatch) -> None: from factory.runners import get_runner_choices @@ -2481,7 +1306,6 @@ def test_includes_all_builtin_runners(self, monkeypatch: pytest.MonkeyPatch) -> metas = get_all_runner_meta() names = {m.name for m in metas} assert "claude" in names - assert "bob" in names def test_handles_runner_without_metadata(self, monkeypatch: pytest.MonkeyPatch) -> None: from factory.runners import get_all_runner_meta @@ -2532,9 +1356,6 @@ def test_includes_builtin_runners(self, monkeypatch: pytest.MonkeyPatch) -> None runners = get_available_runners() assert "claude" in runners - assert "bob" in runners - assert "codex" in runners - assert "opencode" in runners class TestLoadEntrypointRunners: diff --git a/tests/test_session_resume.py b/tests/test_session_resume.py index 557aa8cb7..0ec1d5b48 100644 --- a/tests/test_session_resume.py +++ b/tests/test_session_resume.py @@ -97,13 +97,6 @@ def test_claude_supports_session_resume(self) -> None: meta = ClaudeRunner.metadata() assert meta.supports_session_resume is True - def test_bob_does_not_support_session_resume(self) -> None: - from factory.runners.bob import BobRunner - - meta = BobRunner.metadata() - assert meta.supports_session_resume is False - - class TestClaudeBuildCommandSessionFlags: """Tests for --session-id and --resume flags in build_command.""" diff --git a/tests/test_tmux_cli.py b/tests/test_tmux_cli.py index 081bcbf52..fd52415b6 100644 --- a/tests/test_tmux_cli.py +++ b/tests/test_tmux_cli.py @@ -49,9 +49,7 @@ def test_builds_correct_export_commands(self) -> None: env = { "FACTORY_MODEL": "opus", "ANTHROPIC_API_KEY": "sk-ant-xxx", - "BOBSHELL_API_KEY": "bob-key", "OPENAI_API_KEY": "sk-xxx", - "CODEX_API_KEY": "codex-key", "CLAUDE_CODE_USE_VERTEX": "1", "CLOUD_ML_REGION": "us-central1", "HOME": "/home/user", @@ -106,9 +104,7 @@ def test_builds_correct_export_commands(self) -> None: assert "ANTHROPIC_API_KEY=" in shell_cmd assert "CLAUDE_CODE_USE_VERTEX=" in shell_cmd assert "CLOUD_ML_REGION=" in shell_cmd - assert "BOBSHELL_API_KEY=" in shell_cmd assert "OPENAI_API_KEY=" in shell_cmd - assert "CODEX_API_KEY=" in shell_cmd assert "UNRELATED_VAR" not in shell_cmd assert "HOME=" not in shell_cmd assert "export PATH=" in shell_cmd @@ -128,7 +124,7 @@ def test_propagates_all_flags(self) -> None: focus="dashboard UI", refine="fix login", clean_pr=True, - runner="bob", + runner="claude", prompt="/path/to/spec.md", branch="develop", min_growth=3, diff --git a/tests/test_user_config.py b/tests/test_user_config.py index 999a28cc5..15b31a379 100644 --- a/tests/test_user_config.py +++ b/tests/test_user_config.py @@ -651,8 +651,8 @@ def test_migrate_with_mocked_tomli_w( for key in ( "FACTORY_RUNNER", "FACTORY_MODEL", "FACTORY_PROJECTS_DIR", "FACTORY_VAULT_PATH", "FACTORY_PLAYBOOKS_DIR", "FACTORY_REGISTRY_DIR", - "FACTORY_MANAGED_DIRS", "FACTORY_RUNNER_QUIET", "FACTORY_BOB_DRY_RUN", - "FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE", "FACTORY_CEO_RESPAWN_DISABLED", + "FACTORY_MANAGED_DIRS", "FACTORY_RUNNER_QUIET", + "FACTORY_CEO_RESPAWN_DISABLED", "FACTORY_CEO_MAX_RESPAWNS", ): monkeypatch.delenv(key, raising=False) @@ -755,8 +755,8 @@ def test_migrate_secure_permissions( for key in ( "FACTORY_RUNNER", "FACTORY_MODEL", "FACTORY_PROJECTS_DIR", "FACTORY_VAULT_PATH", "FACTORY_PLAYBOOKS_DIR", "FACTORY_REGISTRY_DIR", - "FACTORY_MANAGED_DIRS", "FACTORY_RUNNER_QUIET", "FACTORY_BOB_DRY_RUN", - "FACTORY_BOB_MAX_INVOCATIONS_PER_CYCLE", "FACTORY_CEO_RESPAWN_DISABLED", + "FACTORY_MANAGED_DIRS", "FACTORY_RUNNER_QUIET", + "FACTORY_CEO_RESPAWN_DISABLED", "FACTORY_CEO_MAX_RESPAWNS", ): monkeypatch.delenv(key, raising=False)