Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
7 changes: 4 additions & 3 deletions CHANGELOG.md
Original file line number Diff line number Diff line change
Expand Up @@ -13,16 +13,17 @@ The first release of Harness² CLI, a general-purpose version of
### Added

- One `harness2` command with subcommands: `init`, `doctor`, `task`, `run`, `status`,
`stop`, `diff`, `eval`, `apply`, `judge`, `report`, `runs`, `config`, `chat`, `guide`,
`stop`, `diff`, `eval`, `apply`, `judge`, `report`, `runs`, `config`, `chat`, `guide`, `login`,
`setup`, `demo`, `bridge`, `completion`.
- Your own tasks: `.harness2/tasks/<id>/task.md` with optional `task.toml`, run in a clean
git worktree of your repository or in a scratch directory, graded optionally by a `check`
command or a `rubric.md`.
- Claude Code and command executors. A project's own harness (CLAUDE.md, `.claude/skills`,
`.claude/agents`, `.claude/scripts`, hooks; or AGENTS.md, `.agents/`, `.codex/hooks.json`)
is imported as the starting harness.
- Subscriptions: `claude-login/<model>` and `chatgpt-login/<model>` use the login of the
installed `claude` or `codex`, with the user's personal setup kept out of the runs.
- Subscriptions: `claude-login/<model>` runs on Claude Code's own login and
`chatgpt-login/<model>` on a Codex login kept for harness2 (`harness2 login codex`), with
personal settings, hooks, plugins and instructions kept out of the runs.
- Any model provider: Anthropic, OpenAI, OpenAI- or Anthropic-compatible servers, Vertex AI
and Bedrock, through provider profiles; `harness2 bridge` serves any of them to Codex.
- Layered settings: defaults, `~/.config/harness2/config.toml`, `.harness2/config.toml`,
Expand Down
8 changes: 6 additions & 2 deletions README.md
Original file line number Diff line number Diff line change
Expand Up @@ -72,7 +72,9 @@ uv tool install git+https://github.com/3ae3ae/harness2-cli
# or: pipx install git+https://github.com/3ae3ae/harness2-cli
```

By default runs use the CLI's own login, so a Claude or ChatGPT subscription is enough. You
By default runs use your subscription, so a Claude or ChatGPT plan is enough: Claude Code
runs on its own login, and Codex on a login kept for harness2 that you make once with
`harness2 login codex`. You
can use an API key or another provider instead; see [Configuration](#configuration). Runs
make many model calls and use money or subscription quota: the [quick start](#quick-start)
shows how to see the number of calls first and how to cap them.
Expand Down Expand Up @@ -138,6 +140,7 @@ harness2 guide --install claude
| `harness2 judge` / `report` | Grade each task's adopted result, and summarise the grades |
| `harness2 runs ls / show / rm` | List, inspect and delete runs |
| `harness2 config show / get / set` | Read and change settings; `show --origin` says where each value comes from |
| `harness2 login` | Check the subscription logins; `login codex` signs Codex in for harness2 |
| `harness2 chat` | Talk to your agent CLI about tasks, runs and results |
| `harness2 guide` | The guide for AI agents that use harness2 |
| `harness2 demo` | A local run with scripted editors and no model calls |
Expand All @@ -161,7 +164,8 @@ channel = "claude"
model = "claude-login/claude-opus-5-5"
```

`chatgpt-login/<model>` uses your ChatGPT subscription through Codex. API keys and other
`chatgpt-login/<model>` uses your ChatGPT subscription through Codex (after
`harness2 login codex`). API keys and other
endpoints work too: Anthropic, OpenAI, any OpenAI-compatible or Anthropic-compatible
server, Vertex AI and Bedrock. To tune an agent other than Claude Code or Codex, the
`command` executor runs your own command and edits the files you list.
Expand Down
7 changes: 7 additions & 0 deletions docs/cli.md
Original file line number Diff line number Diff line change
Expand Up @@ -195,6 +195,13 @@ rubric, or to explain a run's results.
|---|---|
| `--with {claude,codex}` | Which CLI to start. Default: the improver's channel, if installed. |

### `login [codex]`

Without an argument, show whether `claude-login` (Claude Code's own login) and
`chatgpt-login` (the Codex login kept for harness2) are ready. `harness2 login codex` signs
Codex in to ChatGPT for harness2, in its own home, apart from your `~/.codex`; `--device`
uses a device code on machines without a browser.

### `guide`

Print the guide for AI agents that drive harness2, followed by the current project's
Expand Down
28 changes: 20 additions & 8 deletions docs/configuration.md
Original file line number Diff line number Diff line change
Expand Up @@ -52,14 +52,26 @@ all you need:
| Model | Runs on |
|---|---|
| `claude-login/<model>`, e.g. `claude-login/claude-sonnet-5-5` | your Claude subscription, through Claude Code's login (`claude auth login`) |
| `chatgpt-login/<model>`, e.g. `chatgpt-login/gpt-6.1-sol` | your ChatGPT subscription, through Codex's login (`codex login`) |

`harness2 init` picks these by default, and `harness2 doctor` checks that the CLI is logged
in. Runs still keep your personal setup out of the harness being measured: Claude Code runs
with `--setting-sources project,local --strict-mcp-config --no-session-persistence`, so your
own `~/.claude/CLAUDE.md`, skills, plugins, MCP servers and settings are not loaded and no
session history is written; Codex runs with `--ignore-user-config --ignore-rules`. Codex
still reads a global `~/.codex/AGENTS.md` if you have one, and `doctor` says so.
| `chatgpt-login/<model>`, e.g. `chatgpt-login/gpt-6.1-sol` | your ChatGPT subscription, through a Codex login kept for harness2 (`harness2 login codex`, once) |

`harness2 init` picks these by default, and `harness2 login` and `harness2 doctor` say
whether they are ready. Runs keep your personal setup out of the harness being measured:

- **Claude Code** runs on its own login, with `--setting-sources project,local
--strict-mcp-config --no-session-persistence`: your `~/.claude/CLAUDE.md`, skills, plugins,
hooks, MCP servers and settings are not loaded, and no session history is written.
- **Codex** cannot switch off a personal `~/.codex/hooks.json` or `~/.codex/AGENTS.md` with
a flag, so harness2 gives it a home of its own, `~/.local/state/harness2/codex-home`.
`harness2 login codex` signs that home in to ChatGPT once. It is a separate login: your own
`~/.codex` is never read or changed. Runs also pass `--ignore-user-config --ignore-rules`.

To run Codex on your own home anyway, with whatever hooks and instructions it holds, set
its path; `harness2 doctor` then lists the personal files every run will include:

```toml
[providers.chatgpt-login]
home = "~/.codex"
```

Subscriptions have usage limits. When a run reaches one, it stops cleanly (exit code 4),
keeps everything finished so far, and the same `harness2 run` command resumes once the
Expand Down
27 changes: 15 additions & 12 deletions harness2/cli/doctor.py
Original file line number Diff line number Diff line change
Expand Up @@ -16,9 +16,7 @@

from __future__ import annotations

import os
import subprocess
from pathlib import Path

from .. import providers, registry
from ..core.proposer import Proposer
Expand Down Expand Up @@ -130,16 +128,21 @@ def binary_check(binary=binary, channel=channel):
"detail": "; ".join(problems) or detail,
}
)
if any(config.provider(name).type == "chatgpt-login" for name in referenced):
agents = Path(os.environ.get("CODEX_HOME") or Path.home() / ".codex") / "AGENTS.md"
if agents.is_file() and (text := agents.read_text(encoding="utf-8")):
rows.append(
{
"check": "Codex global AGENTS.md",
"ok": True,
"detail": f"your global AGENTS.md ({len(text)} characters) is part of every Codex run",
}
)
if provider.type == "chatgpt-login":
home = providers.codex_login_home(provider)
files = [
str(path)
for name in ("AGENTS.md", "hooks.json")
if (path := home / name).is_file() and path.stat().st_size
]
if files:
rows.append(
{
"check": "Codex home",
"ok": True,
"detail": " and ".join(files) + " are part of every Codex run",
}
)
if args.live:
for (channel, model), spec in sorted(models.items()):

Expand Down
5 changes: 3 additions & 2 deletions harness2/cli/guide.py
Original file line number Diff line number Diff line change
Expand Up @@ -47,8 +47,9 @@
## Workflow

1. `harness2 init --yes [--executor claude|codex] [--auth login|api-key]` creates
.harness2/. `--auth login` uses the agent CLI's own login (a Claude or ChatGPT
subscription). Change any choice afterwards with `harness2 config set KEY VALUE`, for
.harness2/. `--auth login` uses a Claude or ChatGPT subscription: Claude Code's own login
as it is, or for Codex a separate login the person makes once with `harness2 login codex`
(it opens a browser; ask them to run it). `harness2 login --json` shows which are ready. Change any choice afterwards with `harness2 config set KEY VALUE`, for
example `executor.model`.
2. Tasks. `harness2 task new <id> [--files] [--check "<cmd>"]`, then write the request in
.harness2/tasks/<id>/task.md the way the person would really ask for it. Good tasks are
Expand Down
13 changes: 11 additions & 2 deletions harness2/cli/init.py
Original file line number Diff line number Diff line change
Expand Up @@ -37,7 +37,7 @@
PROVIDERS = {"claude": ("claude-login", "anthropic"), "codex": ("chatgpt-login", "openai")}
AUTH = {
"claude": ("your Claude subscription, through Claude Code's own login", "ANTHROPIC_API_KEY"),
"codex": ("your ChatGPT subscription, through Codex's own login", "OPENAI_API_KEY"),
"codex": ("your ChatGPT subscription, through a Codex login for harness2", "OPENAI_API_KEY"),
}

CONFIG = """\
Expand Down Expand Up @@ -264,6 +264,16 @@ def _key_summary(root: Path, models: tuple[str, ...]) -> str:
loaded = S.load(None)
lines = []
for name in dict.fromkeys(providers.split_model(m)[0] for m in models if "/" in m):
if name == "chatgpt-login":
home = providers.codex_login_home(loaded.provider(name))
ok = providers.codex_logged_in(home)
lines.append(
"Codex is signed in for harness2."
if ok
else "Codex is not signed in for harness2 yet: run `harness2 login codex` once "
"(a separate login, so your own ~/.codex settings and hooks stay out of runs)."
)
continue
if name in LOGIN_CHECKS:
argv, cli, fix = LOGIN_CHECKS[name]
ok = shutil.which(argv[0]) and _quiet(argv)
Expand All @@ -289,7 +299,6 @@ def _q(value) -> str:

LOGIN_CHECKS = {
"claude-login": (["claude", "auth", "status"], "Claude Code", "claude auth login"),
"chatgpt-login": (["codex", "login", "status"], "Codex", "codex login"),
}


Expand Down
102 changes: 102 additions & 0 deletions harness2/cli/login.py
Original file line number Diff line number Diff line change
@@ -0,0 +1,102 @@
# Copyright 2026 The Harness2 Authors
#
# Licensed under the Apache License, Version 2.0 (the "License");
# you may not use this file except in compliance with the License.
# You may obtain a copy of the License at
#
# https://www.apache.org/licenses/LICENSE-2.0
#
# Unless required by applicable law or agreed to in writing, software
# distributed under the License is distributed on an "AS IS" BASIS,
# WITHOUT WARRANTIES OR CONDITIONS OF ANY KIND, either express or implied.
# See the License for the specific language governing permissions and
# limitations under the License.

"""Show whether the subscription logins harness2 uses are ready, and set up Codex's.

claude-login uses Claude Code's own login as it is: runs switch off your personal settings,
hooks and plugins with flags. Codex has no such switch for its hooks.json and AGENTS.md, so
chatgpt-login runs use a Codex home of their own, signed in once with `harness2 login codex`.
"""

from __future__ import annotations

import argparse
import os
import shutil
import subprocess
import sys

from .. import providers


def add_parser(sub) -> argparse.ArgumentParser:
p = sub.add_parser(
"login",
help="check the subscription logins, or sign Codex in for harness2",
description="Without an argument, show whether claude-login and chatgpt-login are "
"ready. `harness2 login codex` signs in to ChatGPT in a Codex home kept for harness2, "
"apart from your own ~/.codex.",
)
p.add_argument("service", nargs="?", choices=("codex",), help="sign this CLI in for harness2")
p.add_argument(
"--device",
action="store_true",
help="sign in with a device code, for machines without a browser",
)
return p


def run(args, ctx) -> int:
home = providers.codex_login_home(ctx.settings.provider("chatgpt-login"))
if args.service is None:
rows = _status(home)
ctx.out(rows, lambda data: "\n".join(f"{r['provider']:<14} {r['detail']}" for r in data))
return 0
if shutil.which("codex") is None:
print("`codex` is not on PATH; install Codex first.", file=sys.stderr)
return 1
if not sys.stdin.isatty():
print("Signing in needs a terminal and a browser (or --device).", file=sys.stderr)
return 1
home.mkdir(parents=True, exist_ok=True)
print(
f"Signing Codex in to ChatGPT for harness2. This login lives in {home}, apart from "
"your own ~/.codex, so your personal hooks and AGENTS.md stay out of harness2's runs.\n"
)
env = {k: v for k, v in os.environ.items() if k not in ("OPENAI_API_KEY", "CODEX_API_KEY")}
subprocess.run(
["codex", "login", *(["--device-auth"] if args.device else [])],
env={**env, "CODEX_HOME": str(home)},
)
if not providers.codex_logged_in(home):
print("Codex did not finish signing in.", file=sys.stderr)
return 1
print("Signed in. Models written chatgpt-login/<model> now run on this login.")
return 0


def _status(home) -> list[dict]:
claude = shutil.which("claude") and _ok(["claude", "auth", "status"])
codex = providers.codex_logged_in(home)
return [
{
"provider": "claude-login",
"ready": bool(claude),
"detail": "ready (Claude Code's own login)"
if claude
else "Claude Code is not logged in; run `claude auth login`",
},
{
"provider": "chatgpt-login",
"ready": codex,
"detail": f"ready ({home})" if codex else "not signed in; run `harness2 login codex`",
},
]


def _ok(argv: list[str]) -> bool:
try:
return subprocess.run(argv, capture_output=True, timeout=20).returncode == 0
except (OSError, subprocess.TimeoutExpired):
return False
1 change: 1 addition & 0 deletions harness2/cli/main.py
Original file line number Diff line number Diff line change
Expand Up @@ -39,6 +39,7 @@
"config",
"chat",
"guide",
"login",
"setup",
"demo",
"bridge",
Expand Down
60 changes: 44 additions & 16 deletions harness2/providers.py
Original file line number Diff line number Diff line change
Expand Up @@ -21,9 +21,10 @@
import re
import subprocess
from dataclasses import dataclass
from pathlib import Path
from urllib.parse import urlsplit

from .settings import ConfigError
from .settings import ConfigError, state_dir

PROVIDER_TYPES = ("anthropic", "openai", "vertex", "bedrock", "claude-login", "chatgpt-login")
USAGE_LIMIT = re.compile(
Expand Down Expand Up @@ -53,6 +54,35 @@ class Provider:
api_key_env: str = ""
project: str = ""
region: str = ""
home: str = ""


def codex_login_home(p: Provider) -> Path:
return Path(p.home).expanduser() if p.home else state_dir() / "codex-home"


def codex_logged_in(home: Path) -> bool:
try:
return (
subprocess.run(
["codex", "login", "status"],
env={
**{
k: v
for k, v in os.environ.items()
if k not in (*STRIP_ENV, "OPENAI_API_KEY", "CODEX_API_KEY")
},
"CODEX_HOME": str(home),
},
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
timeout=10,
).returncode
== 0
)
except Exception:
return False


def split_model(ref: str) -> tuple[str, str]:
Expand Down Expand Up @@ -182,22 +212,20 @@ def litellm_params(p: Provider, model: str) -> dict:


def check(p: Provider) -> list[str]:
if p.type in ("claude-login", "chatgpt-login"):
argv = (
["claude", "auth", "status"]
if p.type == "claude-login"
else ["codex", "login", "status"]
)
login = "claude auth login" if p.type == "claude-login" else "codex login"
if p.type == "chatgpt-login":
home = codex_login_home(p)
if codex_logged_in(home):
return []
if p.home:
return [
f"chatgpt-login: not logged in at {home}; run `codex login` with CODEX_HOME={home}"
]
return ["chatgpt-login: not logged in for harness2; run `harness2 login codex`"]
if p.type == "claude-login":
try:
result = subprocess.run(
argv,
env={
k: v
for k, v in os.environ.items()
if k not in STRIP_ENV
and (p.type != "chatgpt-login" or k not in ("OPENAI_API_KEY", "CODEX_API_KEY"))
},
["claude", "auth", "status"],
env={k: v for k, v in os.environ.items() if k not in STRIP_ENV},
stdin=subprocess.DEVNULL,
stdout=subprocess.DEVNULL,
stderr=subprocess.DEVNULL,
Expand All @@ -207,7 +235,7 @@ def check(p: Provider) -> list[str]:
return []
except (OSError, subprocess.SubprocessError):
pass
return [f"{p.type}: not logged in; run `{login}`"]
return ["claude-login: not logged in; run `claude auth login`"]
problems = []
if p.api_key_env and not os.environ.get(p.api_key_env):
problems.append(f"{p.name}: set {p.api_key_env}")
Expand Down
Loading
Loading