From 929113fc695629ad414831b33d73ec4e932146ef Mon Sep 17 00:00:00 2001 From: 3ae3ae Date: Mon, 19 Oct 2026 21:55:22 +0900 Subject: [PATCH 1/2] =?UTF-8?q?docs:=20describe=20Harness=C2=B2=20CLI=20as?= =?UTF-8?q?=20the=20general-purpose=20version=20of=20Harness=C2=B2?= MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The README now presents the fork as Harness² CLI: what it adds to the original release (a comparison table), the run -> diff -> eval -> apply flow around one final harness, subscriptions as the default way to reach models, and how AI agents drive it (--json, --detach, stop, guide). The configuration guide documents claude-login and chatgpt-login, what is and is not isolated, and usage limits; the command reference covers stop, eval, guide, runs rm, --detach, JSON errors and the new run files; How it works explains the final harness step that the CLI adds to the paper's method. Adds a changelog. Co-Authored-By: Claude Opus 5.5 --- CHANGELOG.md | 55 ++++++++ CONTRIBUTING.md | 5 +- README.md | 172 +++++++++++++++----------- docs/cli.md | 63 +++++++--- docs/configuration.md | 48 +++++-- docs/how-it-works.md | 55 +++++--- harness2/cli/bridge.py | 2 +- plugins/harness2-benchmarks/README.md | 4 +- 8 files changed, 287 insertions(+), 117 deletions(-) create mode 100644 CHANGELOG.md diff --git a/CHANGELOG.md b/CHANGELOG.md new file mode 100644 index 00000000..c170ec19 --- /dev/null +++ b/CHANGELOG.md @@ -0,0 +1,55 @@ +# Changelog + +All notable changes to Harness² CLI are documented here. The format follows +[Keep a Changelog](https://keepachangelog.com/en/1.1.0/), and the project uses +[Semantic Versioning](https://semver.org/). + +## [Unreleased] + +The first release of Harness² CLI, a general-purpose version of +[google-research/harness2](https://github.com/google-research/harness2) (forked at +`1cfe254`). The method and its prompts are unchanged; everything around them is new. + +### Added + +- One `harness2` command with subcommands: `init`, `doctor`, `task`, `run`, `status`, + `stop`, `diff`, `eval`, `apply`, `judge`, `report`, `runs`, `config`, `chat`, `guide`, + `setup`, `demo`, `bridge`, `completion`. +- Your own tasks: `.harness2/tasks//task.md` with optional `task.toml`, run in a clean + git worktree of your repository or in a scratch directory, graded optionally by a `check` + command or a `rubric.md`. +- Claude Code and command executors. A project's own harness (CLAUDE.md, `.claude/skills`, + `.claude/agents`, `.claude/scripts`, hooks; or AGENTS.md, `.agents/`, `.codex/hooks.json`) + is imported as the starting harness. +- Subscriptions: `claude-login/` and `chatgpt-login/` use the login of the + installed `claude` or `codex`, with the user's personal setup kept out of the runs. +- Any model provider: Anthropic, OpenAI, OpenAI- or Anthropic-compatible servers, Vertex AI + and Bedrock, through provider profiles; `harness2 bridge` serves any of them to Codex. +- Layered settings: defaults, `~/.config/harness2/config.toml`, `.harness2/config.toml`, + profiles, `HARNESS2_
__` variables and flags, with `config show --origin`. +- One final harness per run, built from every task's adopted harness and lesson; + `harness2 eval` compares it with the starting harness, and `harness2 apply` writes it into + the project, keeping edits made since the run started. +- Run control: `--dry-run` call estimates, `--budget-usd` and `--max-calls` limits, + clean stops at a provider's usage limit, Ctrl-C and `harness2 stop` with resume, + `--detach` background runs, relocatable run directories, `events.jsonl`. +- Use by AI agents: `--json` for every command including errors, flags for every prompt, + and `harness2 guide` (also installable as a Claude Code skill). +- Interactive use: the `init` wizard, `status --watch`, `apply -i`, `--selector human`, + `harness2 chat`. +- CI on Linux and macOS with Python 3.12 and 3.13. + +### Changed + +- The distribution is `harness2-cli`; the command and the import package stay `harness2`. +- The agent CLIs are the ones installed on the system instead of pinned downloads. +- Codex harness scripts live in `.agents/scripts/`, so they never collide with a project's + `scripts/`. + +### Removed + +- The seven `harness2-*` scripts' old flags (`--bench`, `--substrate`); the script names + remain as aliases of the new subcommands. +- The Vertex-only configuration in `.env` from the core package. JobBench, LAB and + WorkBuddy-Bench moved, with that configuration, to the + [harness2-benchmarks plugin](plugins/harness2-benchmarks/). diff --git a/CONTRIBUTING.md b/CONTRIBUTING.md index e4fe97d5..a4e86460 100644 --- a/CONTRIBUTING.md +++ b/CONTRIBUTING.md @@ -1,8 +1,9 @@ # Contributing -Issues and pull requests are welcome. This is an independent fork of +Issues and pull requests are welcome. Harness² CLI is an independent fork of [google-research/harness2](https://github.com/google-research/harness2); changes here are -not sent upstream. +not sent upstream. Every pull request runs the tests and ruff in CI, and notable changes go +in [CHANGELOG.md](CHANGELOG.md) under "Unreleased". ## Development setup diff --git a/README.md b/README.md index ff77c9e2..b33d4a77 100644 --- a/README.md +++ b/README.md @@ -1,9 +1,10 @@

Harness²
- Tune your coding agent's harness on your own tasks + Harness² CLI: tune your coding agent's harness on your own tasks

+ CI Paper Website License @@ -13,22 +14,24 @@ A coding agent is a model plus a **harness**: the instructions, skills, subagent scripts and hooks it runs with. `harness2` improves the harness of the agent you already use (Claude Code's `CLAUDE.md` and `.claude/`, or Codex's `AGENTS.md` and `.agents/`) by running it on tasks from your own project, studying what happened, and refining the harness from -that evidence. The model stays the same. Your files change only when you apply a result you -have reviewed. +that evidence. The model stays the same. Your files change only when you apply the result. -It is a general-purpose command-line tool built on the method of +Harness² CLI is a general-purpose version of [Harness²: Recursive Agent Harnessing for an Open World](https://arxiv.org/abs/2610.14224). -This repository is an independent fork of -[google-research/harness2](https://github.com/google-research/harness2); the paper's -benchmark setup lives on as a [plugin](plugins/harness2-benchmarks/). +The original release ([google-research/harness2](https://github.com/google-research/harness2)) +reproduces the paper on three fixed benchmarks with Vertex AI. This fork turns the same method +into a command-line tool for any project, any task, Claude Code or Codex (or your own agent), +on your subscription or any model provider. [What changed](#what-changed-from-the-original) +lists the differences; the paper's benchmark setup lives on as a +[plugin](plugins/harness2-benchmarks/). - [How it works](#how-it-works) - [Install](#install) - [Quick start](#quick-start) +- [Driving it from an agent](#driving-it-from-an-agent) - [Commands](#commands) - [Configuration](#configuration) -- [Other agents](#other-agents) -- [The paper's benchmarks](#the-papers-benchmarks) +- [What changed from the original](#what-changed-from-the-original) - [Documentation](#documentation) ## How it works @@ -38,13 +41,18 @@ For every task you give it, `harness2`: 1. runs your agent with its current harness and records what it does; 2. has an improver agent propose harness edits from those trajectories, from two angles: how the agent works, and what it needs to know about this kind of task; -3. runs the agent with each candidate harness, and composes a final one from what the +3. runs the agent with each candidate harness, and composes a refined one from what the differences between the runs show; -4. keeps the adopted harness and a short lesson for the next task in the same area. +4. keeps the refined harness and a short lesson for the next task in the same area. -No step needs a grade or a reference answer, so any task you can describe can be used. -If you can also check the result, with a test command or a rubric, `harness2 judge` measures -how much the adopted harness helped. [How it works](docs/how-it-works.md) has the details. +When all tasks are done, it combines what the tasks taught into **one final harness** for +your project. That is the single result you review with `harness2 diff` and write into the +project with `harness2 apply`. + +No step needs a grade or a reference answer, so any task you can describe can be used. If +you can also check results, with a test command or a rubric, `harness2 eval` measures the +final harness against the one you started with. [How it works](docs/how-it-works.md) has +the details.

@@ -54,117 +62,136 @@ how much the adopted harness helped. [How it works](docs/how-it-works.md) has th ## Install -You need Python 3.12 or 3.13, git, and the agent CLI whose harness you want to tune: +You need Python 3.12 or 3.13, git, and the agent CLI whose harness you want to tune, +installed, on your `PATH` and logged in: [Claude Code](https://docs.claude.com/en/docs/claude-code) (`claude`) or -[Codex](https://github.com/openai/codex) (`codex`), installed and on your `PATH`. +[Codex](https://github.com/openai/codex) (`codex`). ```bash -uv tool install git+https://github.com/3ae3ae/harness2 -# or: pipx install git+https://github.com/3ae3ae/harness2 +uv tool install git+https://github.com/3ae3ae/harness2-cli +# or: pipx install git+https://github.com/3ae3ae/harness2-cli ``` -Runs call models through your provider's API, so set the key for the models you choose, -for example `ANTHROPIC_API_KEY` or `OPENAI_API_KEY`. Model calls cost money; the -[quick start](#quick-start) shows how to see the cost of a run before you start it. +By default runs use the CLI's own login, so a Claude or ChatGPT subscription is enough. You +can use an API key or another provider instead; see [Configuration](#configuration). Runs +make many model calls and use money or subscription quota: the [quick start](#quick-start) +shows how to see the number of calls first and how to cap them. ## Quick start Inside the project whose agent setup you want to improve: ```bash -harness2 init # pick the agent and models, creates .harness2/ +harness2 init # pick the agent, how it logs in, where the harness lives harness2 task new fix-login-redirect # then write the request in .harness2/tasks/fix-login-redirect/task.md -harness2 doctor # check CLIs, keys and tasks -harness2 run --dry-run # the task order and how many model calls a run makes -harness2 run # refine the harness on every task +harness2 doctor # check the CLI, its login and the tasks +harness2 run --dry-run # the tasks and how many model calls a run makes +harness2 run --max-calls 200 # refine the harness on every task, then build the final one ``` A run works in temporary git worktrees, so your working tree is never touched. While it -runs, or after it stops: +runs, from another terminal: ```bash harness2 status --watch # progress, spend, the harness picked per task -harness2 judge && harness2 report # grade the results of tasks that have a check or rubric -harness2 diff # what the adopted harness changes -harness2 apply # review it and write it into the project ``` -Run `harness2 run` again to resume an interrupted run; finished steps are reused. Write a -few varied tasks that look like the work you really do: see [Tasks](docs/tasks.md). +When it has finished: + +```bash +harness2 diff # what the final harness changes +harness2 eval # optional: compare it with your current harness on graded tasks +harness2 apply # review the changes and write them into the project +``` + +Running `harness2 run` again resumes an interrupted or limited run and reuses every finished +step. Write a few varied tasks that look like the work you really do: see +[Tasks](docs/tasks.md). + +## Driving it from an agent + +`harness2` is built to be driven by an AI agent as well as by hand. Every command has +`--json`, errors included; no command waits for input without a terminal, since every prompt +has a flag; and long runs go to the background with `harness2 run --detach`, are followed with +`harness2 status --json` and are stopped with `harness2 stop`. + +`harness2 guide` prints the contract for agents, with your project's current state. Tell your +agent to read it, or install it as a Claude Code skill: + +```bash +harness2 guide --install claude +``` ## Commands | Command | What it does | |---|---| -| `harness2 init` | Set up `.harness2/` in a project, with a short interactive wizard | -| `harness2 doctor` | Check the agent CLIs, provider keys, settings and tasks; `--live` also makes one tiny model call per model | +| `harness2 init` | Set up `.harness2/` in a project, with a short wizard (`--yes` for defaults) | +| `harness2 doctor` | Check the agent CLIs, their login or keys, settings and tasks; `--live` also makes one tiny call per model | | `harness2 task new / ls / import` | Create, list and import tasks | -| `harness2 run` | Refine the harness on the tasks; `--dry-run` previews, `--budget-usd` and `--max-calls` cap spending, `--selector human` lets you pick the final harness yourself | +| `harness2 run` | Refine the harness on the tasks and build the final harness; `--dry-run` previews, `--max-calls` / `--budget-usd` cap spending, `--detach` runs in the background, `--selector human` lets you pick between candidates | | `harness2 status` | A run's progress; `--watch` keeps it updated | -| `harness2 judge` | Grade the result each task adopted | -| `harness2 report` | Summarise grades, per run | -| `harness2 runs ls / show` | List runs and inspect one | -| `harness2 diff` | Show how an adopted harness differs from the starting one | -| `harness2 apply` | Write an adopted harness into the project, after showing the changes; `-i` to accept file by file | +| `harness2 stop` | Stop a running run; running it again resumes | +| `harness2 diff` | What the final harness changes | +| `harness2 eval` | Compare the final harness with the starting one on graded tasks | +| `harness2 apply` | Write the final harness into the project after showing the changes; `-i` to accept file by file | +| `harness2 judge` / `report` | Grade each task's adopted result, and summarise the grades | +| `harness2 runs ls / show / rm` | List, inspect and delete runs | | `harness2 config show / get / set` | Read and change settings; `show --origin` says where each value comes from | | `harness2 chat` | Talk to your agent CLI about tasks, runs and results | +| `harness2 guide` | The guide for AI agents that use harness2 | | `harness2 demo` | A local run with scripted editors and no model calls | -Every command takes `--help`. See the [command reference](docs/cli.md) for all flags, exit +Every command takes `--help`. The [command reference](docs/cli.md) lists every flag, the exit codes and the layout of a run directory. ## Configuration -Settings come from `~/.config/harness2/config.toml`, the project's -`.harness2/config.toml`, `HARNESS2_*` environment variables and flags, in that order of -precedence. Models are written `provider/model`; any OpenAI-compatible or -Anthropic-compatible endpoint, Vertex AI and Bedrock can be used: +Settings come from `~/.config/harness2/config.toml`, the project's `.harness2/config.toml`, +`HARNESS2_*` environment variables and flags, in that order of precedence. Models are +written `provider/model`: ```toml [executor] name = "claude" -model = "anthropic/claude-sonnet-5-5" +model = "claude-login/claude-sonnet-5-5" # your Claude subscription, through Claude Code [improver] channel = "claude" -model = "anthropic/claude-opus-5-5" - -[providers.local] # e.g. a vLLM or LM Studio server -type = "openai" -base_url = "http://localhost:8000/v1" -api_key_env = "LOCAL_API_KEY" +model = "claude-login/claude-opus-5-5" ``` -[Configuration](docs/configuration.md) lists every setting. +`chatgpt-login/` uses your ChatGPT subscription through Codex. API keys and other +endpoints work too: Anthropic, OpenAI, any OpenAI-compatible or Anthropic-compatible +server, Vertex AI and Bedrock. To tune an agent other than Claude Code or Codex, the +`command` executor runs your own command and edits the files you list. +[Configuration](docs/configuration.md) covers every setting. -## Other agents +## What changed from the original -To tune an agent other than Claude Code or Codex, use the `command` executor: give it the -command that runs one task and the files that make up its harness. +| | google-research/harness2 | Harness² CLI | +|---|---|---| +| Tasks | JobBench, LAB, WorkBuddy-Bench | your own tasks, in your repository or from files; the benchmarks as a plugin | +| Agents | OpenCode, Codex, pinned builds | Claude Code, Codex or any command, as installed on your system | +| Models | Vertex AI | your Claude or ChatGPT subscription, or any API provider | +| Result | one adopted harness per task, kept for scoring | one final harness for the project, which you review and apply | +| Interface | seven scripts, settings in `.env` | one `harness2` command, layered TOML settings, JSON output, background runs, resumable limits | -```toml -[executor] -name = "command" -command = "my-agent --task {prompt_file} --workdir {workspace}" -editable = ["prompts/*.md", "agent.yaml"] -``` - -## The paper's benchmarks - -JobBench, LAB and WorkBuddy-Bench, the benchmarks of the paper, are packaged as the -[harness2-benchmarks plugin](plugins/harness2-benchmarks/). Install it next to `harness2` -and they become suites: `harness2 run --suite jb`. +The method itself (propose, probe, compose, and the library of lessons) and its prompts are +the paper's. [How it works](docs/how-it-works.md) describes what was added around it. ## Documentation -- [How it works](docs/how-it-works.md): the method, what a run costs, what it stores +- [How it works](docs/how-it-works.md): the method, the final harness, what a run costs and stores - [Tasks](docs/tasks.md): writing tasks, workspaces, grading -- [Configuration](docs/configuration.md): settings, models and providers +- [Configuration](docs/configuration.md): settings, subscriptions, models and providers - [Command reference](docs/cli.md): every command and flag -- [Contributing](CONTRIBUTING.md) +- [Changelog](CHANGELOG.md) and [contributing](CONTRIBUTING.md) ## Citation +If you use Harness² CLI in research, cite the paper: + ```bibtex @article{xu2026harness2, title = {Harness$^2$: Recursive Agent Harnessing for an Open World}, @@ -178,5 +205,6 @@ and they become suites: `harness2 run --suite jb`. ## License Apache 2.0; see [LICENSE](LICENSE). The method and the original implementation are by the -paper's authors at Google; this fork is not a Google product and is not maintained by -Google. Third-party code in the benchmarks plugin keeps its upstream licenses. +paper's authors at Google; Harness² CLI is an independent fork, not a Google product, and is +not maintained by Google. Third-party code in the benchmarks plugin keeps its upstream +licenses. diff --git a/docs/cli.md b/docs/cli.md index 425cb9f9..983305c3 100644 --- a/docs/cli.md +++ b/docs/cli.md @@ -23,7 +23,7 @@ These go before the command: `harness2 -C ../api --json status`. | `-c KEY=VALUE`, `--config KEY=VALUE` | Override one setting for this command; repeatable. See [Configuration](configuration.md). | | `-q`, `--quiet` | Print results only, no progress. | | `-v`, `--verbose` | Print debug details. | -| `--json` | Print results as JSON on stdout; progress still goes to stderr. | +| `--json` | Print results as JSON on stdout; progress still goes to stderr. A failing command prints `{"error": "...", "exit_code": n}` instead of an error line. | | `--version` | Print the version. | ## Commands @@ -31,13 +31,15 @@ These go before the command: `harness2 -C ../api --json status`. ### `init` Set up harness2 in a project: asks which agent runs your tasks (`claude`, `codex` or a -custom `command`), which models to use, and where the harness lives; writes +custom `command`), whether it reaches its models through the CLI's own login (your +subscription) or an API key, which models to use, and where the harness lives; writes `.harness2/config.toml`, `.harness2/.gitignore` and `.harness2/tasks/`. | Option | Meaning | |---|---| -| `--yes`, `-y` | Take every default without asking. | +| `--yes`, `-y` | Take every default without asking. Without a terminal, defaults are taken anyway. | | `--executor {claude,codex,command}` | Skip the executor question. | +| `--auth {login,api-key}` | Skip the login question. | | `--force` | Overwrite an existing config file. | ### `doctor` @@ -63,8 +65,10 @@ makes a files task (with a `files/` directory) instead of a project task. `impor ### `run` -Refine the harness on the tasks. Running the same command again resumes the run and reuses -every finished step. +Refine the harness on the tasks, then combine what the tasks taught into one final harness +per recursion mode (one more improver session each), stored in the run's `consolidated/` +directory. Running the same command again resumes the run and reuses every finished step; +if building the final harness failed, only that step runs again. | Option | Meaning | |---|---| @@ -84,11 +88,19 @@ every finished step. | `--selector human` | Pick the final harness yourself when `k` > 1. | | `--steer TEXT\|@FILE` | Guidance added to every improver session. | | `--dry-run` | Show the tasks and the number of model calls; run nothing. | +| `--detach` | Start the run in the background and return at once with its name, pid and log file (`run.log` in the run directory). | | `--split-file FILE` | Task partition file, for benchmark suites. | A run keeps the settings it started with. Resuming it with a different executor, model, improver or task list is refused, listing what changed; use a new `--name` for a new -experiment. The limits apply to the whole run, across resumes. +experiment. The limits apply to the whole run, across resumes. A run that is already going +is not started twice. + +### `stop [NAME]` + +Stop a running run, whether it was started with `--detach` or in another terminal. It +finishes the model calls in flight and exits; running the same `harness2 run` command again +resumes it. ### `status [NAME]` @@ -113,26 +125,40 @@ rubric or the judge model changes. Average the grades per recursion mode. Several names compare runs. -### `runs ls` and `runs show NAME` +### `runs ls | show NAME | rm NAME` -List runs with their progress and spend, or show one run's settings, totals and tasks. +List runs with their progress and spend, show one run's state and settings, or delete a run +and everything it stored (`--yes` to skip the question). ### `diff [NAME]` -Show how the harness a run adopted for a task differs from the harness the run started -from. +Show how the run's final harness differs from the harness the run started from. | Option | Meaning | |---|---| -| `--task ID` | The task whose adopted harness to show. Default: the last task the run finished. | +| `--task ID` | Show the harness adopted for this one task instead of the final harness. | | `--mode {parallel,sequential}` | The recursion mode to take it from (default parallel). | | `--stat` | List the changed files only. | +### `eval [NAME]` + +Run the agent with the starting harness and with the final harness on the run's tasks that +have a check or rubric, grade both, and compare the pass rates. Costs one agent run per +task, harness and repeat. + +| Option | Meaning | +|---|---| +| `--repeat N` | Runs per task and harness (default 1). More runs, steadier numbers. | +| `--split {all,train,test}` | Compare on one split, for example the `test` tasks the run did not use. | +| `--only-tasks ID,...` | Compare on these tasks only. | +| `--mode {parallel,sequential}` | Which mode's final harness to compare (default parallel). | + ### `apply [NAME]` -Write an adopted harness into the project. Only what the run changed relative to the +Write the run's final harness into the project. Only what the run changed relative to the harness it started from is applied, so edits you made since are kept; a file changed both by the run and in the project is skipped unless `--force`. The changes are listed first. +With `--json`, `--yes` or `--dry-run` is required, since nothing is asked. | Option | Meaning | |---|---| @@ -169,6 +195,12 @@ rubric, or to explain a run's results. |---|---| | `--with {claude,codex}` | Which CLI to start. Default: the improver's channel, if installed. | +### `guide` + +Print the guide for AI agents that drive harness2, followed by the current project's +settings, tasks and runs. `--install claude` installs it as a user-level Claude Code skill +(`~/.claude/skills/harness2/`), which harness2's own runs never load. + ### `setup SUITE [ARGS...]` Run a plugin suite's installer, for example `harness2 setup lab` for the @@ -183,7 +215,7 @@ Run the method on two toy tasks with scripted editors: no model calls, a few sec Serve a configured model as an OpenAI Responses API endpoint through LiteLLM, so the Codex executor or channel can use providers it cannot reach directly. Needs -`pip install 'harness2[bridge]'`. +`pip install 'harness2-cli[bridge]'`. | Option | Meaning | |---|---| @@ -206,7 +238,7 @@ Print a shell completion script; see [below](#shell-completion). | 1 | An error: bad settings, a missing CLI or key, a failed check, a run that could not continue. | | 2 | Invalid command-line usage. | | 3 | `run` finished, but some tasks failed. | -| 4 | `run` stopped at `--budget-usd` or `--max-calls`. Raise the limit and rerun to resume. | +| 4 | `run` or `eval` stopped at `--budget-usd`, `--max-calls` or the provider's usage limit. Rerun to resume. | | 130 | Interrupted with Ctrl-C. Rerun to resume. | ## A run on disk @@ -214,7 +246,10 @@ Print a shell completion script; see [below](#shell-completion). ``` .harness2/runs// run.json settings, task list and schedule the run started with + run.pid, run.log while it runs: its process, and its output when started with --detach events.jsonl one line per agent run, model session, selection and task, with costs + consolidated// the final harness, the improver's notes and what it was built from + eval// results of harness2 eval base_harness/ the project's harness when the run started rollouts// one agent run: prompt.md, stream.jsonl (trajectory), final.md, deliverable.patch or output/, meta.json, judge.json diff --git a/docs/configuration.md b/docs/configuration.md index d576c19f..f2757184 100644 --- a/docs/configuration.md +++ b/docs/configuration.md @@ -42,8 +42,33 @@ parse as TOML is taken as a plain string: `-c executor.model=openai/gpt-6.1-sol` ## Models and providers Every model is written `provider/model`. The provider part names a `[providers.]` -table, which says what kind of endpoint it is and where its key comes from. Two providers -are built in: +table, which says what kind of endpoint it is and how it is reached. + +### Subscriptions + +Two built-in providers use the login your installed CLI already has, so a subscription is +all you need: + +| Model | Runs on | +|---|---| +| `claude-login/`, e.g. `claude-login/claude-sonnet-5-5` | your Claude subscription, through Claude Code's login (`claude auth login`) | +| `chatgpt-login/`, e.g. `chatgpt-login/gpt-6.1-sol` | your ChatGPT subscription, through Codex's login (`codex login`) | + +`harness2 init` picks these by default, and `harness2 doctor` checks that the CLI is logged +in. Runs still keep your personal setup out of the harness being measured: Claude Code runs +with `--setting-sources project,local --strict-mcp-config --no-session-persistence`, so your +own `~/.claude/CLAUDE.md`, skills, plugins, MCP servers and settings are not loaded and no +session history is written; Codex runs with `--ignore-user-config --ignore-rules`. Codex +still reads a global `~/.codex/AGENTS.md` if you have one, and `doctor` says so. + +Subscriptions have usage limits. When a run reaches one, it stops cleanly (exit code 4), +keeps everything finished so far, and the same `harness2 run` command resumes once the +limit resets. With a subscription, keep `run.phase_width` and `run.proposer_concurrency` +low, and cap runs with `--max-calls`. + +### API keys and other endpoints + +Two API-key providers are built in: ```toml [providers.anthropic] @@ -88,6 +113,8 @@ Each agent CLI speaks to some provider types and not others: | Provider type | Claude Code (`claude`) | Codex (`codex`) | OpenCode (`opencode`) | |---|---|---|---| +| `claude-login` | yes | no | no | +| `chatgpt-login` | no | yes | no | | `anthropic` | yes | via `harness2 bridge` | yes | | `openai` | no | yes (Responses API) | yes (Chat Completions) | | `vertex` | yes | via `harness2 bridge` | yes | @@ -95,7 +122,7 @@ Each agent CLI speaks to some provider types and not others: `harness2 doctor` reports a model whose provider its CLI cannot use. To run Codex on a model it cannot reach directly, serve that model as a Responses API endpoint with the bridge -(`pip install 'harness2[bridge]'`), then point an `openai`-type provider at it: +(`pip install 'harness2-cli[bridge]'`), then point an `openai`-type provider at it: ```bash harness2 bridge --model anthropic/claude-sonnet-5-5 @@ -104,12 +131,10 @@ harness2 bridge --model anthropic/claude-sonnet-5-5 The agent CLIs are the ones installed on your system (`claude`, `codex`, `opencode` on `PATH`). Set `executor.bin` or `improver.bin` to use another binary or a full command. -**Logins are not used.** Every run gives the agent CLI a fresh, empty configuration -directory (`CLAUDE_CONFIG_DIR` for Claude Code, `CODEX_HOME` for Codex), so your personal -instructions, skills, plugins and settings never leak into the harness being measured. -That also means the CLI does not see your interactive login: models are reached with the -provider's API key. `harness2 chat` is the exception: it starts your own CLI as you normally -use it. +With an API-key provider, every run also gets a fresh, empty configuration directory +(`CLAUDE_CONFIG_DIR` for Claude Code, `CODEX_HOME` for Codex), the strictest isolation. +`harness2 chat` is different from runs: it starts your own CLI exactly as you normally use +it. ## Reference @@ -169,8 +194,9 @@ person. | `max_calls` | `0` | Stop the run after this many model calls (rollouts plus sessions). `0` means no limit. | Spending is counted from the costs the CLIs report. Claude Code and OpenCode report dollars; -Codex reports tokens only, so `budget_usd` does not see Codex spending. Use `max_calls` to -bound a Codex run. +Codex reports tokens only, so `budget_usd` does not see Codex spending, and with a +subscription the reported dollars are an estimate rather than a charge. Use `max_calls` to +bound those runs. A call already in flight when a limit is reached still finishes. ### `[suite]` diff --git a/docs/how-it-works.md b/docs/how-it-works.md index 106e7680..cc32aee2 100644 --- a/docs/how-it-works.md +++ b/docs/how-it-works.md @@ -5,13 +5,16 @@ files the agent reads and runs around every task. For Claude Code those are `CLA skills, subagents, helper scripts and hooks; for Codex, `AGENTS.md`, skills, scripts and hooks. The method is described in the paper [Harness²: Recursive Agent Harnessing for an Open World](https://arxiv.org/abs/2610.14224). -This page explains what a run does, what it costs, and what you get back. +This page explains what a run does, what it costs, and what you get back, including the +parts Harness² CLI adds to the paper's method: one final harness per run, `eval` and +`apply`. - [The harness components](#the-harness-components) - [One task: propose, probe, compose](#one-task-propose-probe-compose) - [Scaling with k](#scaling-with-k) -- [Picking the final harness](#picking-the-final-harness) +- [Picking each task's harness](#picking-each-tasks-harness) - [Carrying lessons forward](#carrying-lessons-forward) +- [One harness for the project](#one-harness-for-the-project) - [What a run gives you](#what-a-run-gives-you) ## The harness components @@ -42,8 +45,8 @@ For each task, a run: works on *what a practitioner knows* about this kind of task. Each writes a candidate harness. 3. **Probes** every candidate by running the agent with it on the same task. -4. **Composes** a final harness from the contrast between all those runs: which edits changed - behaviour, and how. The final harness is run too. +4. **Composes** a refined harness from the contrast between all those runs: which edits + changed behaviour, and how. The refined harness is run too. No step looks at a grade. The improvers see trajectories and results, and decide from the differences between runs what helped. This is what lets Harness² work on tasks that have no @@ -67,15 +70,16 @@ otherwise dropped. Per task and mode, a run makes **4k agent runs** (including the `k` runs of the current harness, which both modes share) and **3k improver sessions**, plus one reflector session that writes the task's lesson and, when `k` is more than 1, one selector session that picks -the final. `harness2 run --dry-run` prints the exact counts for your tasks before anything is -spent, and `--budget-usd` / `--max-calls` stop a run at a limit. +the task's harness. Building the final harness adds one session per run and mode. +`harness2 run --dry-run` prints the exact counts for your tasks before anything is spent, +and `--budget-usd` / `--max-calls` stop a run at a limit. -## Picking the final harness +## Picking each task's harness -With `k` = 1 there is one final. With more, a **selector** compares the finals' results -blind: it sees the task and each result under an anonymous id, never which harness produced -it or how that harness was built, and picks the best. That pick is the task's **adopted** -harness. +With `k` = 1 there is one composed harness per task. With more, a **selector** compares the +composed harnesses' results blind: it sees the task and each result under an anonymous id, +never which harness produced it or how that harness was built, and picks the best. That pick +is the task's **adopted** harness. You can make the pick yourself with `harness2 run --selector human`: the run stops at each selection and shows you the anonymous candidates in the terminal. @@ -91,6 +95,26 @@ the harness that improves harnesses gets better as tasks go by. Tasks in a domain run one after another so the library can grow; separate domains run in parallel (`--workers`). Each run has its own library, so experiments never mix. +## One harness for the project + +In the paper, every task ends with its own adopted harness: the method adapts the agent to +the task in front of it, and the adopted harnesses are kept for scoring and as examples for +later tasks. A project needs one harness that it can keep. So when all tasks are done, +`harness2 run` adds one step of its own: an improver session reads the starting harness, +every task's request, the harness adopted for it and the lesson recorded, and composes **one +final harness** for the project's next tasks. It keeps what helped on several tasks or what +a lesson supports as general, and drops facts tied to a single task. The result is checked +against the same rules as every other proposal. + +That final harness is what you work with: + +- `harness2 diff` shows how it differs from the harness you started with; +- `harness2 eval` runs the agent with both on the tasks that have a check or rubric and + compares the pass rates, the evidence for whether it actually helps; run it on `test` + tasks the run did not use for the fairest comparison; +- `harness2 apply` writes it into the project, applying only what the run changed, so edits + you made meanwhile are kept, and touching nothing but the harness files. + ## What a run gives you Everything is stored under `.harness2/runs//`: @@ -101,8 +125,9 @@ Everything is stored under `.harness2/runs//`: lesson. - `rollouts/`: per agent run, the trajectory, the result and the final message. - `library/`: the adopted harnesses and lessons per domain. +- `consolidated/`: the final harness and the improver's notes on why each change is there. -Use `harness2 status` while it runs, `harness2 judge` and `harness2 report` to grade the -results, `harness2 diff` to read what changed in the harness, and `harness2 apply` to write an -adopted harness into your project. Running the same `harness2 run` command again resumes -where it stopped, reusing every finished step. +Use `harness2 status` while it runs, `harness2 diff`, `harness2 eval` and `harness2 apply` on +the final harness, and `harness2 judge` / `harness2 report` to grade each task's adopted +result. Running the same `harness2 run` command again resumes where it stopped, reusing every +finished step. diff --git a/harness2/cli/bridge.py b/harness2/cli/bridge.py index 7a0879f7..f82222e4 100644 --- a/harness2/cli/bridge.py +++ b/harness2/cli/bridge.py @@ -68,7 +68,7 @@ def add_parser(subparsers): help="serve a model as a Responses API endpoint for Codex", description="Serve any configured provider/model as an OpenAI Responses API " "endpoint through LiteLLM, so Codex can run on it. Needs the bridge extra: " - "pip install 'harness2[bridge]'.", + "pip install 'harness2-cli[bridge]'.", ) ) diff --git a/plugins/harness2-benchmarks/README.md b/plugins/harness2-benchmarks/README.md index b8532135..df458aec 100644 --- a/plugins/harness2-benchmarks/README.md +++ b/plugins/harness2-benchmarks/README.md @@ -2,7 +2,7 @@ The three benchmarks of the paper [Harness²: Recursive Agent Harnessing for an Open World](https://arxiv.org/abs/2610.14224), -packaged as suites for the `harness2` command: +packaged as suites for the `harness2` command of [Harness² CLI](../../README.md): | Suite | Benchmark | Work | |---|---|---| @@ -15,7 +15,7 @@ compatibility patches of the paper, and grade with each benchmark's own judge. ## Install -From a checkout of this repository, next to `harness2`: +From a checkout of this repository, next to `harness2-cli`: ```bash uv pip install -e . -e 'plugins/harness2-benchmarks[render]' From 225d18e38ea90b2b0a4cefa280eab3ae87370bcb Mon Sep 17 00:00:00 2001 From: 3ae3ae Date: Mon, 19 Oct 2026 21:59:15 +0900 Subject: [PATCH 2/2] feat(status): say whether a run is still going `harness2 status` (and its --json) now reports whether the run's process is alive and its pid, and says when a run stopped without finishing, for example after being killed, so the user knows to resume it. Co-Authored-By: Claude Opus 5.5 --- harness2/cli/status.py | 9 +++++++++ 1 file changed, 9 insertions(+) diff --git a/harness2/cli/status.py b/harness2/cli/status.py index 7059e902..7a55b273 100644 --- a/harness2/cli/status.py +++ b/harness2/cli/status.py @@ -106,7 +106,10 @@ def snapshot(run_dir: Path) -> dict: if row["state"] == "running": row["state"] = "stopped" plan = meta.get("schedule", {}) + process = C.run_process(run_dir) return { + "active": process is not None, + "pid": process["pid"] if process else None, "run": meta.get("name", run_dir.name), "suite": meta.get("suite", ""), "executor": meta.get("executor", ""), @@ -126,6 +129,8 @@ def render(snap: dict) -> str: head = f"{snap['run']} · {snap['suite']} suite · {snap['executor']}" if snap["k"]: head += f" · k={snap['k']} {snap['modes']}" + if snap.get("active"): + head += f" · running (pid {snap['pid']})" last = snap["last_event"] lines = [ head, @@ -156,6 +161,10 @@ def render(snap: dict) -> str: lines.append(f"run {'finished' if status == 'ok' else 'ended: ' + status}{why}{tail}") elif last: lines.append(f"latest: {_describe(last)}") + if not snap.get("active"): + lines.append( + "not running: it stopped without finishing; run the same command to resume" + ) return "\n".join(lines)