Agent Benchmark Lab

English | 简体中文

An evidence-backed laboratory for evaluating real coding-agent combinations:

harness x model x fixed task cohort x environment x budget profile
    -> raw evidence + multi-dimensional scores + uncertainty

What It Is

Agent Benchmark Lab helps answer two different practical questions:

• Current-configuration comparison: Which of Claude Code, opencode, Codex, Aider, Antigravity CLI, MimoCode, Grok, or another CLI works best with the model each tool is currently configured to use?
• Strict same-model comparison: Which harness is stronger only after every adapter can select, and its output verifies, the same model identity?

It is not a generic model leaderboard. It does not disguise outcomes from different evaluators as one universal score.

Current Maturity

The project can run isolated tasks, public and hidden tests, protected-file integrity checks, real CLI adapters, repeated task/suite/matrix experiments, resumable runs, reports, and separate local versus official SWE-bench result tracks.

It cannot yet honestly claim that one harness is generally stronger than another. That requires at least three identified real configurations, three repetitions each, and empirical task-discrimination evidence from screening-report. The current selection-ready task count is intentionally zero.

comprehensive-screening-v1 is the fixed full cohort:

• 11 local comparative tasks spanning expert to easy.
• 9 hard SWE-bench Verified ranking candidates.
• 1 diagnostic tail task, visible but excluded from official ranking rate.

It reports three complementary views:

1. Local scorecard: ten local dimensions, verified coverage, domain axes, mean, variance, and confidence intervals.
2. Official SWE resolution track: official resolved / scorable attempts and resolution rate.
3. balanced-v1 decision index: 55% local verified-normalized score + 45% official SWE resolution rate. It is a transparent personal selection aid, not an objective benchmark total.

Official evaluator errors are not model failures. Official tasks never enter the local total, domain axes, radar charts, or matrix ranking.

The current launcher simplifies running, but it still assumes a prepared developer environment. Automatic prerequisite repair and a standalone native macOS app are planned future work; this repository does not yet claim a zero-setup experience.

Documentation

Audience	Document	Purpose
Users	User guide	Setup, smoke runs, matrices, full cohort, statistics, recovery, and result interpretation.
Developers and future agents	Developer guide	Architecture, task/adapter/scorer/report extensions, tests, recovery, and maintenance rules.
Result reviewers	Benchmark readiness audit	Known scientific limits and the path to higher-confidence conclusions.
Maintainers	Handoff and next-agent prompt	Current status, hard rules, and unfinished work.
Scoring readers	Scoring specification and capability taxonomy	Weights, evidence status, statistics, and domain axes.
Corpus contributors	Corpus strategy and task provenance	Local tasks, SWE-bench, Terminal-Bench, and licensing boundaries.

Quick Start

Install browser-verification prerequisites once:

python3 -m pip install -r requirements-browser.txt
npm ci
npx playwright install chromium

Check the local environment without spending model tokens:

PYTHONPATH=src python3 -m agent_benchmark.cli.main doctor
PYTHONPATH=src python3 -m agent_benchmark.cli.main validate
PYTHONPATH=src python3 -m agent_benchmark.cli.main audit
PYTHONPATH=src python3 -m agent_benchmark.cli.main list-adapters
PYTHONPATH=src python3 -m agent_benchmark.cli.main list-suites

Run a low-cost real-adapter smoke test:

PYTHONPATH=src python3 -m agent_benchmark.cli.main run \
  --task python-bugfix --adapter claude-code \
  --model unspecified --repetitions 1

--model unspecified means “use this CLI's current default model and record any observed identity.” It compares current full configurations, not a same-model harness experiment.

One Friendly Command

On macOS, install the short command once from the repository:

./benchmark install

Then create a folder for one experiment, enter it, and run one short command:

mkdir -p ~/Documents/claude-benchmark-2026-07
cd ~/Documents/claude-benchmark-2026-07
agent-benchmark claude-code

It runs doctor and preflight, uses the fixed full cohort with three repetitions, writes every artifact into the current directory, refreshes ./dashboard/index.html, and opens the dashboard when finished. This command always uses the harness's current default model; it is the normal path. Claude Code and OpenCode default runs each perform one isolated structured identity probe before task 1; OpenCode reads its model from the temporary session export. Antigravity CLI can run through agy --print and accepts --model, but its current print mode does not expose reliable default-model or structured telemetry, so the report keeps that evidence pending rather than guessing. The TUI is a Rich-managed, full-width alternate-screen run surface rather than a hand-written ANSI cursor loop; it shows current work, verified model identity, progress, context, and real lifecycle/workspace activity. Recoverable state is saved in suite-*/live_status.json. A smaller cohort remains agent-benchmark opencode hard-discrimination; explicit --model, --suite, and --repetitions options are documented in the user guide as advanced overrides.

Run The Full Fixed Cohort

Run the no-cost preflight first:

PYTHONPATH=src python3 -m agent_benchmark.cli.main preflight-matrix \
  --suite comprehensive-screening-v1 \
  --adapters claude-code --models unspecified \
  --budget-profiles stress --repetitions 3

Then run the cohort:

PYTHONPATH=src python3 -m agent_benchmark.cli.main run-suite \
  --suite comprehensive-screening-v1 \
  --adapter claude-code --model unspecified \
  --budget-profile stress --repetitions 3 \
  --label comprehensive-v1

This can take a long time and consume real provider budget. Resume an interrupted suite from the directory printed by the command:

PYTHONPATH=src python3 -m agent_benchmark.cli.main resume-suite \
  --suite-run-dir runs/<suite-run-id>

Reports are saved as suite_report.html, suite_report.md, and suite_summary.json. Build the historical dashboard with:

PYTHONPATH=src python3 -m agent_benchmark.cli.main dashboard
open runs/dashboard/index.html

Rules That Protect Result Quality

• Do not rank from one run, an unknown model identity, or low evidence coverage.
• Use a fixed cohort and at least three repetitions for serious comparisons; inspect means, variance, task-level 95% CI, failure samples, and domain axes.
• Read strict total, verified coverage, and verified-normalized score together.
• Tool-call count is not cost. Raw token/cost data is saved, but cost efficiency needs a task budget frozen before the experiment.
• external_frozen metadata is not an imported benchmark task. SWE-bench and Terminal-Bench must use their official bridges/evaluators.
• runs/ contains ignored evidence artifacts. Do not commit model logs, patches, credentials, or sensitive outputs.

License

This project is licensed under the GNU General Public License v3.0 (GPL-3.0). You may use, modify, and distribute it; when distributing a derivative work, provide corresponding source under GPL-3.0 and retain the required notices. The project is provided without warranty.

---

简体中文

这是一个用来评估真实 coding-agent 组合的、保留完整证据的长期实验室：

harness x model x 固定题目组 x 环境 x 预算档位
    -> 原始证据 + 多维分数 + 不确定性

它解决什么问题

项目区分两个实际问题：

• 当前配置比较：此刻的 Claude Code、opencode、Codex、Aider、Antigravity CLI、MimoCode、Grok 等 CLI 各自使用当前默认模型时，哪个更适合实际工作流？
• 严格同模型比较：只有每个 adapter 都能选择同一个模型，且运行输出验证模型身份后，才比较 harness 本身。

它不是通用模型排行榜，也不会把不同 evaluator 的结果伪装成一个万能总分。

当前成熟度

当前版本已经能完成：隔离任务、公开/隐藏测试、受保护文件完整性检查、真实 CLI adapter、重复 task/suite/matrix 实验、中断恢复、报告，以及本地题与官方 SWE-bench 结果分轨。

但它还不能诚实地宣称“某个 harness 普遍更强”。要得到这样的结论，至少需要三个模型身份可识别的真实配置、每个配置三次重复，并通过 screening-report 证明题目确实具有区分度。当前 selection_ready 题数为 0，这是严谨状态提示，不是运行故障。

comprehensive-screening-v1 是固定完整卷：

• 11 道从 expert 到 easy 的本地比较题；
• 9 道 SWE-bench Verified 高难候选题；
• 1 道只展示、不进入官方排名率的诊断尾题。

它会给出三种互补结果：

1. 本地 scorecard：本地十维、verified coverage、领域轴、均值、方差和置信区间。
2. 官方 SWE resolution track：官方 evaluator 的 resolved / scorable attempts 与修复率。
3. balanced-v1 决策指数：55% 本地 verified-normalized 分 + 45% 官方 SWE 修复率。它是透明的个人选型辅助，不是客观 benchmark 总分。

官方 evaluator 错误不算模型失败；官方题不会进入本地总分、领域轴、雷达图或 matrix 排名。

当前启动器已经简化运行，但仍假定使用者准备好了开发环境。自动修复前置环境与独立原生 macOS App 都是后续计划；本仓库目前不宣称“零配置即可使用”。

文档入口

读者	文档	用途
使用者	用户手册	环境配置、smoke、矩阵、完整卷、统计、恢复和结果解释。
开发者与下一位 agent	开发者手册	架构、Task/Adapter/Scorer/Report 扩展、测试、恢复与维护规则。
审查结果的人	基准就绪审计	已知科学边界与提高可信度的路径。
维护者	交接文档 与 下一 agent 提示词	当前状态、硬规则和未完成事项。
理解分数的人	评分规范 与 能力分类	权重、证据状态、统计和领域轴。
题库贡献者	题库策略 与 题目来源	本地题、SWE-bench、Terminal-Bench 与许可证边界。

快速开始

首次使用浏览器视觉检查时安装依赖：

python3 -m pip install -r requirements-browser.txt
npm ci
npx playwright install chromium

先做不花模型额度的环境和项目检查：

PYTHONPATH=src python3 -m agent_benchmark.cli.main doctor
PYTHONPATH=src python3 -m agent_benchmark.cli.main validate
PYTHONPATH=src python3 -m agent_benchmark.cli.main audit
PYTHONPATH=src python3 -m agent_benchmark.cli.main list-adapters
PYTHONPATH=src python3 -m agent_benchmark.cli.main list-suites

用真实 adapter 进行一次低成本 smoke：

PYTHONPATH=src python3 -m agent_benchmark.cli.main run \
  --task python-bugfix --adapter claude-code \
  --model unspecified --repetitions 1

--model unspecified 表示“使用该 CLI 此刻默认模型，并记录实际观察到的模型身份”。因此它比较的是当前完整配置，而不是同模型 harness 对决。

一条友好命令

在 macOS 上，先在仓库根目录安装一次短命令：

./benchmark install

之后新建本次实验的文件夹，进入它，只运行一条短命令：

mkdir -p ~/Documents/claude-benchmark-2026-07
cd ~/Documents/claude-benchmark-2026-07
agent-benchmark claude-code

它会自动执行 doctor 和 preflight，使用三重复的固定完整卷，把所有原始证据和结论写入当前目录，刷新 ./dashboard/index.html，并在结束后自动打开看板。这个默认命令始终使用 harness 当前配置的默认模型，才是正常使用路径。Claude Code 与 OpenCode 的默认模型实验都会先运行一次隔离的结构化身份探针；OpenCode 会从临时 session export 中读取实际模型，因此全屏 TUI 会在第一道题前显示已观察到的模型，而不是只写“默认”。Antigravity CLI 已通过 agy --print 接入，并支持显式 --model；但当前 print 输出不提供可靠的默认模型和结构化遥测，报告会保持 pending，绝不猜测。终端页使用 Rich 管理的全宽 alternate-screen 运行界面，不再手写 ANSI 光标刷新；它展示当前任务、已验证模型、进度、运行上下文与真实生命周期/工作区活动。可恢复状态保存在 suite-*/live_status.json；较小题组仍可用旧写法 agent-benchmark opencode hard-discrimination。显式 --model、--suite、--repetitions 仅是用户手册里的高级覆盖参数。

运行固定完整卷

先运行不花 token 的预检：

PYTHONPATH=src python3 -m agent_benchmark.cli.main preflight-matrix \
  --suite comprehensive-screening-v1 \
  --adapters claude-code --models unspecified \
  --budget-profiles stress --repetitions 3

然后执行完整卷：

PYTHONPATH=src python3 -m agent_benchmark.cli.main run-suite \
  --suite comprehensive-screening-v1 \
  --adapter claude-code --model unspecified \
  --budget-profile stress --repetitions 3 \
  --label comprehensive-v1

它可能运行很久并消耗真实 provider 额度。中断后使用命令打印出的目录恢复：

PYTHONPATH=src python3 -m agent_benchmark.cli.main resume-suite \
  --suite-run-dir runs/<suite-run-id>

报告在 suite_report.html、suite_report.md 和 suite_summary.json。生成历史看板：

PYTHONPATH=src python3 -m agent_benchmark.cli.main dashboard
open runs/dashboard/index.html

保护结果质量的规则

• 不要根据单次 run、未知模型身份或低 evidence coverage 排名。
• 严肃比较使用固定题目组与至少三次重复；同时看均值、方差、任务级 95% CI、失败样本和领域轴。
• strict total、verified coverage、verified-normalized score 必须一起看。
• 工具调用数不是成本。原始 token/cost 会保存，但成本效率必须在实验前冻结任务级预算后才能计分。
• external_frozen 元数据不是已导入题；SWE-bench 与 Terminal-Bench 必须走官方 bridge/evaluator。
• runs/ 保存被忽略的证据产物；不要提交模型日志、patch、凭据或敏感输出。

许可证

本项目使用 GNU General Public License v3.0（GPL-3.0）。你可以使用、修改和分发；发布基于本项目的衍生作品时，须按 GPL-3.0 提供相应源码并保留必要声明。项目按“现状”提供，不提供担保。