[feat] Run benchmark suites as workflows

Parent: #1137

Goal

Labels from sessions and workflow runs show how agents did on whatever work came along. They cannot say whether a different tool or model would do better on the same work, because the work differs from run to run. A benchmark answers that: the same tasks, from the same starting commit, judged by the same check, run once per arm and repeated.

A benchmark is a teamai feature built on the workflow system like any other run: a default workflow, teamai-benchmark, shipped with teamai and replaceable by the team, plus a teamai benchmark command for the parts that are about benchmarks. Every driver works it through next and report.

This issue adds suites with arms that change the tool or the model. Arms that change the team's assets, or run one of the team's workflows, are #1181.

Files

In the team repo, next to workflows/:

# benchmarks/auth-fixes/suite.yaml
schemaVersion: v1alpha1
name: auth-fixes
description: Bugs from the auth module, fixed by the implementer role
repo: https://github.com/acme/api          # the project the tasks start from
defaults: { tool: claude, model: strong }  # model is optional; use an alias when arms change the tool
arms:
  - name: baseline
  - name: codex
    tool: codex
  - name: fast
    model: fast
repetitions: 3                             # 1 to 10, default 3
timeout: 6h                                # required, the whole run's
parallel: 2                                # optional, 1 to 32, default 1: lanes baselines and trials run in
setup: npm ci                              # optional, runs in each fresh clone before the agent and the check
scorers: [diff-quality]                    # optional, scorers that judge every trial
trial: { timeout: 8m }                     # optional, limits of each trial's prompt (#1400)

# benchmarks/auth-fixes/tasks/expired-token.yaml
schemaVersion: v1alpha1
title: Reject expired refresh tokens
prompt: |
  Refresh tokens past their expiry are still accepted. Fix it.
startingCommit: 3f2c9a7d1e0b4c6a8f5d2e1b0c9a8f7e6d5c4b3a   # full 40-character commit of repo
check:
  run: npx vitest run test/auth/refresh.test.ts
  protect: ['test/**', 'vitest.config.ts', 'package.json', 'package-lock.json']
  add: [test/auth/refresh.test.ts]                          # written from source.fix before the check; never in the agent's clone
source: { platform: claude-code, sessionId: 4b1e..., fix: 9a1d4e7c2b8f0a3d6e5c1b9f8a7d2e4c6b0a1f3e }

A task has exactly one of check and successCriteria. successCriteria is text a correctness judge reads, for work no command can decide. Suites are placed by namespace like workflows: benchmarks/<suite>/ is shared, benchmarks/<ns>/<suite>/ reaches only members with <ns> active.

Validation, at pull, reported per file like workflows:

1 to 6 arms with unique names. Each arm changes at most one of the defaults keys, and at most one arm changes none.
repetitions is a whole number from 1 to 10, and parallel a whole number from 1 to 32 (must be a whole number from 1 to 32).
A model name in defaults.model while arms change the tool is a warning: an alias resolves per tool, a name does not.
startingCommit is a full commit. check.run is non-empty, and check.protect is a non-empty list of paths relative to the repo. A task without check has successCriteria.
timeout is set.
A suite with no valid task is not cached. The last valid version is kept, as for workflows.
teamai review --branch reports the same diagnostics as fail findings, for every namespace's suites.

Running

Under the shipped strict sandbox, every trial prompt has permissions: unattended, including prompts in arm workflows. The local driver uses the tool's unattended mode. An external driver declares can.unattended and honors the lease's permissions. A team override without a strict sandbox omits the key and reports that trials use the tool's permission settings. A tool with no supported unattended mode fails tool_unavailable.

$ teamai benchmark run auth-fixes --sandbox
$ teamai benchmark run auth-fixes --sandbox --member-aliases        # resolve arm aliases with ~/.teamai/models/aliases.yaml over the team's
$ teamai benchmark run auth-fixes --sandbox --fail-on-regression    # in CI: also exit 1 when an arm got worse
$ teamai benchmark try ../team/benchmarks/auth-fixes/tasks/expired-token.yaml

Command	What it does
`teamai benchmark run <suite> [--driver local\|skill] [--sandbox] [--member-aliases] [--fail-on-regression]`	Plans the suite once without starting anything, so a suite that cannot run leaves no run. Then starts the `teamai-benchmark` workflow with `--inputs '{"suite":"<suite>","aliases":"team"}'` (`"member"` with `--member-aliases`) and `--timeout <suite timeout>`, on the local driver unless `--driver skill`. Prints the `done` envelope on stdout and the results on stderr.
`teamai benchmark try <task file>`	Tries one task's check at `startingCommit` and at `source.fix`, before the task joins the suite. No agent, no run, no sandbox.
`teamai benchmark plan <suite> [--aliases team\|member]`	A step of the workflow: freezes the run's plan and prints `{ definition, plan }`. Refuses a run with no timeout, or one longer than the suite's.
`teamai benchmark check --baseline \| --trial`	A step of the workflow, in a trial's clone: restores `protect` (trial only), writes `add`, runs the check, prints `{"result":"passed"\|"failed"\|"could_not_run"}` and exits 0. Exits 1 when it could not set the check up.
`teamai benchmark report`	A `finally` step of the workflow: computes the results, writes the benchmark record, prints `{ text, table }`.

All five verbs appear in teamai benchmark --help and the generated commands.md. People and CI use run and try; the workflow calls the other three. An external driver starts the same workflow itself and works it through next and report:

$ teamai workflow start teamai-benchmark --inputs '{"suite":"auth-fixes","aliases":"team"}' --timeout 6h

The default workflow

# workflows/teamai-benchmark.yaml, shipped in the npm package; a team file of the same name overrides it
schemaVersion: v1alpha1
name: teamai-benchmark
description: Run a benchmark suite. Every arm works the same tasks in fresh clones, judged by checks it never sees.
invoke: deny
inputs: { suite: string, aliases: string }
steps:
  - id: plan
    command: teamai benchmark plan "$SUITE" --aliases "$ALIASES"
    env: { SUITE: "{{ inputs.suite }}", ALIASES: "{{ inputs.aliases }}" }
    output: { definition: workflow, plan: json }
  - id: trials
    needs: [plan]
    workflow: { from: plan }
    requires: { isolation: sandbox, strict: true }
finally:
  - id: report
    command: teamai benchmark report
    output: { text: string, table: json }
outputs: { text: steps.report.outputs.text, table: steps.report.outputs.table }

teamai benchmark run auth-fixes
├─ dry plan          the same freeze as the plan step, with no run: a refusal leaves no run
└─ workflow start teamai-benchmark --inputs '{"suite":"auth-fixes","aliases":"team"}' --timeout 6h
   ├─ plan           command: reads the suite, tasks, aliases and scorers at the pulled team revision, fills the
   │                 commit cache, prints a workflow with one step per baseline and per trial
   ├─ trials         runs that workflow; strict sandbox for every step beneath it
   │  ├─ baselines.<task>                    clone at startingCommit → setup → teamai benchmark check --baseline
   │  └─ r<rep>.<task>.<arm>                 clone at startingCommit, team installed → setup
   │     ├─ prompt                           the arm's tool, model and effort
   │     └─ check                            teamai benchmark check --trial
   └─ finally: report                        runs whatever the run ended as: succeeded, failed, timed out or cancelled

With parallel: n, each trial sits under its lane, lane<k>.r<rep>.<task>.<arm>, and the trials of a lane run one after another.

invoke: deny: an agent never starts it on its own; people and CI start it with teamai benchmark run.

A team's own `teamai-benchmark` workflow

A team file workflows/teamai-benchmark.yaml replaces the default, as teamai-share-learnings is replaced. For example, a team whose runners are not declared sandboxes drops strict:

  - id: trials
    needs: [plan]
    workflow: { from: plan }
    requires: { isolation: sandbox }        # best-effort: no credential is withheld, and isolation is recorded unenforced

teamai benchmark run names the team file it runs, on stderr: Running the team's workflows/teamai-benchmark.yaml, which replaces teamai's benchmark workflow.
It refuses a team teamai-benchmark workflow without the input suite before it creates a run (invalid_inputs): The team's workflows/teamai-benchmark.yaml replaces teamai's benchmark workflow and takes no input suite, so teamai benchmark run cannot give it the suite. Add inputs: { suite: string } to it, or remove it to run teamai's default.
A team workflow without the input aliases plans with the team's aliases. With --member-aliases, run refuses it the same way, naming --member-aliases.
The override is placed by namespace like any team workflow: workflows/<ns>/teamai-benchmark.yaml replaces the default for members who receive <ns>, and run names that file. Two active namespaces that both define it cause workflow_namespace_conflict at pull and at run start. The run refuses rather than falling back to the shipped workflow.
The benchmark record carries the workflow's name, origin (default or team) and sourceHash, so results from two definitions are never read as one.

What each trial gets

A trial gets
A fresh clone at `startingCommit`	teamai prepares it at lease (`next --prepare`), offline, from the checkout's objects or the commit cache. Its own git data, no remotes, hooks off. The member's checkout is never changed.
The team assets	Installed in the clone at the team revision this checkout last pulled, frozen when `plan` runs. A pull during the run changes nothing it runs.
The arm's tool, model and effort	Written into the trial's prompt step as `tool:` and the exact `model: { name, effort }`, resolved from the alias when `plan` runs. Every driver passes them as given.
An environment	Under the default's strict sandbox, without `GITHUB_TOKEN`, `GH_TOKEN`, `GITLAB_TOKEN`, `GL_TOKEN`, `CI_JOB_TOKEN`, `SSH_AUTH_SOCK`, `GIT_ASKPASS` and `SSH_ASKPASS`, for every process teamai starts: setup, the tool and the check.
The member's home directory	As it is. teamai cannot remove its logins, MCP servers and plugins, which is why trials need a sandbox. "Hidden" means not in the clone: an agent running as the same user can still read the machine.

The sandbox is declared by whoever runs it: --sandbox on teamai benchmark run, or workflows.sandbox: true in config. Without either, the default's strict requires cannot be met on the local driver, so teamai benchmark run refuses before it creates a run (requirement_unmet):

Step "trials" requires strict isolation, and the local driver here is not a sandbox. Run it in CI or a container, and declare it with --sandbox or workflows.sandbox: true.

Where a benchmark cannot pin the team, it does not run. plan refuses every suite with team_ref_blocked, before a run is created, in single-repo mode, where user scope also reaches the workspace (Q19), and for a team served over HTTP. In single-repo mode the message is:

Benchmarks do not run in single-repo mode: the project's own commit decides which team assets a checkout has, so no arm can run on a team revision of its own. Start the benchmark from a checkout whose team has a team repo of its own.

Trials. One per task × arm × repetition, ordered by repetition, then task in file name order, then arm. In repetition r, counting from 0, the arms are rotated by r (A B C, then B C A, then C A B), so no arm always runs first. A failed or unscored trial never skips another. With parallel: n, baselines and trials run in n lanes; the effective number is the smaller of n and the driver's own limit (the local driver's workflows.max_parallel, 1 unless set). Trials that run together compete for the machine, so their durations and costs interfere.

Each check is tried first. Before a task's trials, setup and the check run once in a clean clone at startingCommit. If the check passes there, the task is reported invalid: check passes without changes and its trials do not run. If setup fails, or the check cannot start, times out or is killed by a signal, the task is reported check could not run and its trials are not_run. Exit codes 129 to 192 count as a check that could not run, outside Windows.

Correctness judge. A task with successCriteria is judged by a scorer plan generates for it, teamai:correctness:<task>: labels correct (score 1) and incorrect (score 0). It and the suite's scorers judge every trial (sampling: all), on the diff from startingCommit, commits included, and the prompt's session. The run's judges settle before report runs.

Drivers. A step goes only to a driver that can run it:

The step needs	The driver declares
a clone prepared by teamai	`clone` in `can.workspace`, and asks with `next --prepare`
the strict sandbox	`can.isolation: sandbox` and `can.env: true`
the arm's tool	the tool in `can.tools`
setup in the clone	`can.setup`
the `plan`, `check` and `report` command steps	nothing more: it runs each with the lease's `env`, which carries `TEAMAI_WORKFLOW_ATTEMPT`, in the lease's workspace

On the local driver, a trial whose tool it cannot run fails tool_unavailable; report counts it not_run and prints Arm "<arm>" uses <tool>, which the local driver cannot run, so its trials did not run. The other arms still run.

A step that runs teamai … uses the CLI that runs the step, put first on its PATH, so npx users and a second global install run the same version.

Isolated run. A run that opens a clone place is isolated: it is never published or claimed, never retried with workflow retry, publishes no session, takes no snapshot, counts in no member stats and adds no labels to the Scorers page. Its workflow run record carries isolated: true.

Results

$ teamai benchmark run auth-fixes --sandbox > done.json          # the results on stderr, sample numbers
| Arm | Tool | Model (source) | Planned | Passed | Failed | Error | Unscored | Not run | Stopped by a limit | Deviations | Not verified | Pass rate |
| --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- | --- |
| baseline | claude | claude-opus-5-5 high (team) | 9 | 4 | 2 | 0 | 0 | 3 | 0 | 0 | 0 | 44% (4/9) |
| codex | codex | <strong for codex> (team) | 9 | 3 | 3 | 0 | 0 | 3 | 0 | 0 | 0 | 33% (3/9) |
| fast | claude | claude-haiku-5-5 (team) | 9 | 2 | 4 | 0 | 0 | 3 | 1 | 0 | 0 | 22% (2/9) |

| Arm | Median tokens (input + output) | Median duration | Cost |
| --- | --- | --- | --- |
| baseline | 48210 (6) | 212.4 s (6) | 3.10 USD (6) |
…

| Task | baseline | codex | fast |
| --- | --- | --- | --- |
| expired-token | 67% (2/3) | 33% (1/3) | 33% (1/3) |
…

| Task | Baseline |
| --- | --- |
| old-bug | invalid: check passes without changes |

These results hold for these tasks, this run and this machine.

$ jq -r '.data.done.outputs.text // empty' done.json              # the same text, e.g. for a CI job summary

Result	When
`passed`	The check exits 0, or the correctness label passed
`failed`	The check ran to the end and failed, or the correctness label did not pass
`error`	Preparing the clone, setup or the prompt failed (tool error, a timeout or budget that was not a trial limit), or the check could not start or finish
`unscored`	No check, and the correctness judge gave no label
`not_run`	The trial never ran: no driver for its tool, an invalid task, or the run timed out or was cancelled first

The table. Per arm: intended tool and model with their source (team, or member under --member-aliases), a Team column when an arm's team is not the pulled one ([feat] Compare team asset sets in benchmarks #1181), planned trials, each result, trials stopped by a limit, deviations, not verified and pass rate (passed / planned trials); then median tokens, median duration and cost as the tool stated them, each with how many values it used; then each task's pass rate per arm; then the tasks whose baseline did not let their trials run. Before the tables, one line for each arm whose tool the driver could not launch. After them, record: pending while the record waits for publication. report prints it as Markdown pipe tables, so it renders in a CI job summary, and the done envelope carries it to every driver in outputs.text and outputs.table. benchmark run prints the envelope on stdout and the text on stderr.
A deviation is a trial whose observed agent contradicts its arm on the tool, the team revision or a concrete model id, or that lists a suite limit in unenforced. A missing value is not verified.
The record. report writes a benchmark record to workflows/runs/<run id>/benchmark.json in the scope's data directory, which the dashboard's Benchmarks page reads. With usage reporting on, it also queues it for delivery, which publishes it to benchmarks/runs/<run id>.json on teamai-reports and retries on later deliveries. With it off, the record stays in the run directory only. It holds the suite and task hashes, the frozen baseline team revision, the scorers' rubric hashes, the arms and every trial with its result, limit, unenforced, duration, tokens and cost, and the run's outcome. No prompts, commands, credential URLs, local paths or hidden check bytes. A reader rebuilds the table from the record alone, for example each arm's pass rate, in a checkout of teamai-reports:
```
$ jq -r '. as $b | $b.arms[] | .name as $arm | [$b.trials[] | select(.arm == $arm)] | (map(select(.result == "passed")) | length) as $passed | "\($arm) \($passed / length * 100 | round)% (\($passed)/\(length))"' benchmarks/runs/<run id>.json
```
A record not yet published shows as record: pending.
The workflow run record no longer carries the benchmark; it only says isolated: true.

Exit code of teamai benchmark run:

The run	`report`	`--fail-on-regression`	Exit
succeeded, whatever the trials scored	succeeded	not set	0
failed, cancelled or timed out (as `workflow start`)	any	any	1
succeeded	failed or cancelled	any	1
succeeded	succeeded, record pending	not set	0
succeeded	succeeded	an arm's drop is separated from the comparable record	1
succeeded	succeeded	no comparable record	0, with a notice

report is a finally step, so a run that times out or is cancelled still gets its table and record, with the trials it never reached as not_run.

--fail-on-regression compares the run's record with the newest earlier record published on teamai-reports whose run ended succeeded and ran the same suite, the same tasks by sourceHash, and the same arms: name, tool, model, effort, workflow and team choice, never the revision a team resolved to. This machine's pending queue is left out: a record not yet on the reports branch is not history. Trials pair by task and repetition, as the Compare tab pairs them, and an arm drops when the whole 95% interval of its paired difference is below zero. It prints one of these on stderr (sample numbers):

Arm "codex" passed 2/9 paired trials, against 6/9 in run r1a2b3c4d5e6f7a8b: a drop of 44 points, with a 95% interval of -71 to -10 points.
No arm's pass rate dropped by more than chance against run r1a2b3c4d5e6f7a8b.
No earlier succeeded run of suite auth-fixes ran the same tasks on the same arms on teamai-reports: recording is off, or nothing comparable has been published yet. --fail-on-regression has nothing to compare with.

Task fix, `check.add` and `try`

# benchmarks/auth-fixes/tasks/startup.yaml
check:
  run: npx vitest run test/startup.test.ts
  protect: ['test/**']
  add: [test/startup.test.ts]                # written from source.fix; the agent never sees it
source: { platform: claude-code, sessionId: 4b1e..., fix: 9a1d4e7c2b8f0a3d6e5c1b9f8a7d2e4c6b0a1f3e }

$ teamai benchmark try ../team/benchmarks/auth-fixes/tasks/startup.yaml

source.fix is an optional full commit in the task's source: the commit that fixed the task. check.add lists repo files under protect, one path each, no glob, whose content comes from source.fix (or a { from, to } file of the suite directory, #1401). Every check, in a run and in try, prepares its clone in one order: clone at the commit, setup, restore protect (a trial's check only), write add, run the check. So the baseline writes them after setup too. The bytes come from the commit cache, never from the clone, so the agent cannot read them from git. Pull and review --branch refuse a glob, a path no protect entry matches, and a string add entry without source.fix.

teamai benchmark try <task file> reads the task file and the suite.yaml two directories up from it as written, validates both as pull does, and needs no teamai init. It runs in a checkout of the suite's repo, fetching a missing commit by SHA from that repo into a temporary cache. It creates no run and no record. It prints its progress on stderr (Task startup: trying the check at 3f2c9a7) and one envelope on stdout: {"data":{"try":{"task","message"}}} with exit 0, or {"error":{"code":"task_try_failed","message"}} with exit 1.

Task	At `startingCommit`	At `source.fix`	Exit	Message
startup	fails	passes	0	`Task "startup": the check fails at 3f2c9a7 and passes at the fix 9a1d4e7, with the protected paths as 3f2c9a7 has them.`
no-fix	fails	no `source.fix`	0	`Task "no-fix": the check fails at 3f2c9a7, as the task expects. The task has no source.fix, so the check was not tried on a fix.`
old-bug	passes	not tried	1	`Task "old-bug": the check passes without changes at 3f2c9a7, so it cannot tell a fix from no change. Pick a startingCommit where the check fails, or fix the check.`
bad-fix	fails	fails	1	`Task "bad-fix": the check fails at the fix 9a1d4e7 (exit 1), with the protected paths as 3f2c9a7 has them.` When the fix changes a protected file that `check.add` does not list, it adds: `The fix changes test/startup.test.ts, which protect puts back as 3f2c9a7 has them: list a test the fix brought in check.add.`
add-missing	could not be tried	not tried	1	`Task "add-missing" could not be tried at 3f2c9a7: check.add path test/missing.test.ts is not a file at the fix 9a1d4e7c2b8f0a3d6e5c1b9f8a7d2e4c6b0a1f3e. List files the fix has in check.add, or fix source.fix.`
typo	could not run	not tried	1	`Task "typo": the check "npx vitset run" could not run at 3f2c9a7 (not_found). Make the command runnable after setup there, or fix check.run.`
criteria-only	no check	no check	0	`Task "criteria-only" is judged by successCriteria, so there is no check to try. Its files are valid.`

add is written at both commits, so a missing add path stops the try at startingCommit, after setup. In a run, the same path fails the check step with trial_prepare_failed and the same message: the baseline is check could not run, and a trial is error.

Acceptance criteria

Pull reports a suite with 7 arms, a suite without timeout, a suite with parallel: 33, an arm that changes both tool and model, a task with both check and successCriteria, a task with an empty protect, and a startingCommit that is not a full commit, each with its file.
In a declared sandbox, teamai benchmark run of a fixture suite with 2 tasks, 2 arms and 2 repetitions runs 8 trials. The record lists all 8 with task, arm, repetition and order, and the arm order of repetition 2 is the reverse of repetition 1. When the first trial fails, the other 7 still run, and the run succeeds.
Every trial's agent.project.commit is the task's startingCommit, its clone has no remotes and its own git data, and git status of the member's checkout is the same before and after the run. Every trial's agent.team.revision is the frozen baseline team revision, also when a team commit is pushed, and the member's checkout pulls it, during the run. A checkout whose team revision is not on the remote fails before any trial.
Inside a trial, env in setup, the prompt and the check shows none of the variables listed above.
A trial that edits vitest.config.ts to skip the test, or adds a helper under test/, is checked against the restored files and fails.
A task whose check passes on startingCommit is reported invalid and none of its trials run. A baseline whose setup fails, or whose check cannot start, times out or is killed, is reported check could not run, and its trials are not_run. A check that times out after a successful prompt gives error, not failed.
In a model arm, each trial's agent.model.requested is the model plan resolved for the arm, on every driver, also when models/aliases.yaml changes during the run. In a tool arm, each trial's agent.tool.name is the arm's tool. An arm whose alias has no model for its tool fails before a run is created.
A trial whose reported model contradicts a concrete requested id counts as a deviation and stays in the denominator. A trial with no reported model is not verified.
A successCriteria task has a teamai:correctness:<task> label on every trial. A trial whose judge fails is unscored, not passed.
Without --sandbox or workflows.sandbox: true, teamai benchmark run refuses with requirement_unmet and the message above, and creates no run. In a declared sandbox, an arm whose tool the local driver cannot run has its trials not_run with the Arm "<arm>" uses <tool> message, the other arms' trials run, and the run succeeds. One passed trial out of ten planned shows a pass rate of 10%.
Editing the suite and pulling during a run does not change that run's plan. After removing the suite from the cache, the printed numbers can still be computed from the record with the jq command above, which the guide gives verbatim. The record keeps the baseline team revision and every scorer's rubric hash even when observations are missing, a scorer failed, or every trial is not_run.
A fake external driver that calls teamai workflow next --prepare and report, declares can.isolation: sandbox, can.env: true, can.unattended: true and clone in can.workspace, and lists only claude in can.tools, runs a one-task suite with repetitions: 1, a Claude arm and a Codex arm. It runs the Claude trial and its check, is never leased the Codex trial, and the suite's timeout ends the run with the Codex trial not_run. The done envelope that next then returns carries the table in outputs. A driver that does not declare the sandbox or unattended mode, or does not list the tool, gets no trial lease.
A branch holding the 7-arm suite from criterion 1 fails teamai review --branch with exit 1 and a fail finding for that file.
teamai benchmark run exits as the table under "Results" says. A run cancelled during its trials still prints the table, with the trials it never reached not_run.
A team workflows/teamai-benchmark.yaml without strict runs instead of the default, teamai benchmark run names it, and the record's workflow.origin is team. So does a workflows/<ns>/teamai-benchmark.yaml for a member whose role receives <ns>. A team teamai-benchmark workflow without the input suite is refused before a run is created, and one without the input aliases is refused under --member-aliases.
A benchmark run is refused by workflow retry, workflow publish and --claim, publishes no session and counts in no member stats. Its workflow run record has isolated: true and no benchmark field.
With parallel: 2 and workflows.max_parallel: 2, two trials run at once; with workflows.max_parallel unset, one at a time.
With --fail-on-regression, an arm whose paired pass rate drops with its whole interval below zero against the newest comparable published record exits 1, naming the arm and that run. A comparable record still in this machine's pending queue is not compared with. With no comparable record it exits 0 with the notice above.
teamai benchmark try gives each row of the table above, with its exit code and envelope, with no --sandbox and no teamai init.
In single-repo mode, and where user scope also reaches the workspace, teamai benchmark run of any suite refuses team_ref_blocked and creates no run.
With --member-aliases, an arm whose alias the member's ~/.teamai/models/aliases.yaml maps runs on that model, and the table and record show member as its source. teamai benchmark plan --aliases with a value other than team or member is refused, naming both.

Definition of done

docs/guide/benchmarks.md and docs/guide/zh-CN/benchmarks.md: teamai benchmark run|try, the default workflow and a team override, what a trial gets, the exit codes, parallel, --member-aliases, --fail-on-regression, the refusals in single-repo mode and under user scope, what an external driver needs, and the CI jobs below.
The CI jobs for GitHub Actions and GitLab CI install a pinned teamai version and keep tokens from trial agents: GitHub with persist-credentials: false and permissions: contents: read, GitLab by stripping CI_JOB_TOKEN from the origin URL.
docs/guide/workflow.md and its zh-CN page list benchmark among the default workflows. The workflow guide pages document the generic keys this adds: finally, workflow: { from }, the workflow and json output types, done.outputs, workflow start --timeout, workspace: { clone }, tool:, model: { name, effort }, on_limit: end_turn, requires.strict, permissions: unattended, can.unattended and scorers.sampling: all.
docs/dev/workflow-engine.md: the engine, the protocol and the drivers never import src/benchmark/, enforced by workflow-module-boundaries.test.ts; a teamai feature is a default workflow plus domain verbs.
skill-data/core/references/commands.md regenerated with npx vitest run commands-reference -u; tool-support regenerated with npx vitest run tool-support -u; the README agent matrix's benchmarks column in every README*.md.
CHANGELOG.md under ## [Unreleased], in its features list: teamai benchmark and the benchmark default workflow.
Real-CLI record in the PR: teamai benchmark run of a one-task suite with two arms on Claude in a sandbox, plus teamai benchmark try of that task.

Test plan

Parser tests for criterion 1 and the generic keys.
Unit tests for plan (trial list, rotation, lanes, frozen values), report (classification, table, record) and the exit codes.
E2E through the built CLI with a fixture project and a fake tool for criteria 2 to 6, 10 to 12 and 14 to 21.
One real-CLI run for the PR body: a one-task suite with two arms on Claude in a container or CI job, plus Codex if available.

Watch out for

The engine and the drivers must not learn about benchmarks. Anything a benchmark needs from them is a generic key that any workflow can use.
A suite's defaults sets its baseline arm. It is not a workflow's defaults, and the guide pages do not present them as one key.
Keep the suite's timeout below the host's job limit, or the job is killed before report prints the table.

Out of scope

Arms that change team assets or run a team workflow ([feat] Compare team asset sets in benchmarks #1181).
Turning a past session into a task ([feat] session-to-task: turn a session's bug fix into a benchmark task #1402).
A recommendation of an arm. The table and the Benchmarks page show results and intervals only.
Moving other teamai features onto default workflows (Run teamai's own features as default workflows #1589).

Dependencies

Notes from #1175 and #1085 for benchmark runs:

A trial needs its own base. Step 1 says agent configuration is "the starting commit's, as [feat] Add workflow workspaces, branches, and session continuation #1080 makes it for every step", while "Watch out for" says [feat] Add workflow workspaces, branches, and session continuation #1080 uses the run's base. feat(workflow): add workspace branches #1175 does the latter. WorkspaceContract already carries base: { ref, commit } and configCommit on each lease (workspace.ts:23-30), but planWorkspace fills both from run.base (workspace.ts:248, :271-272). This issue would add the clone mode and fill those fields per trial. The trial lease is already gated by can.trials, so can.workspace needs no new value; an older engine rejects unknown can.workspace values (engine.ts:1119-1121). The lease parser requires a string base.ref today (local-driver.ts:321); F2 in [feat] Close the #1080 gaps that #1175 leaves open #1246 makes ref optional, so a trial can leave it out. Criterion 3 then also asserts that the trial's AGENTS.md/CLAUDE.md equal the starting commit's version when the member's differ.
A failed check is a measurement, not a run failure. A workflow fails when any step stopped (engine.ts:719), so a failed check ends the benchmark failed and counts in member stats (records.ts:527-528). I'd keep how the benchmark ended apart from how its trials scored: a failed check does not fail the run, a cancellation or a run limit still ends it as today, and every planned trial stays in the table, not_run included. The record's top-level workflow, sourceHash and teamRevision are not defined for a benchmark; define them, and leave benchmark runs out of the workflows totals in stats and teamai digest (records.ts:504-531, digest.ts:480-483).
Duration has no source. Nothing records time for a step: settledAt is a revision (step-state.ts:111), and no report or lease field holds a time. Lease-to-report time would include the clone, init, the pinned pull and setup. I'd have the driver report the time around the tool's launch as a report field on the trial's prompt step. A trial's tokens and duration are named as the prompt step's, without setup, check or scorer work.
Tokens do not compare across tools, so record cost when the tool states it. The arm table compares "median tokens", and the headline example puts a Claude arm against a Codex arm; different tokenizers and prices make that ranking misleading. A report could carry cost: { amount, unit } next to tokens when the tool's own output states a cost, with unit as the tool gives it, never estimated from a price table. Record the prompt's cost and the scoring cost separately (failed judge attempts included, see [feat] Add workflow scorers and recorded quality labels #1085), compare arms only in the same unit and over the same tasks, mark a total incomplete when a part is unknown, and never count a missing cost as zero. I have not checked which tools print a cost; OpenCode's session export carries a numeric cost (src/__tests__/fixtures/opencode/export-tool-calls.json:10). The same field gives the Workflow Runs page a cost per run, and with [feat] Deliver workflow branches as pull requests #1090's prs, a cost per PR.
The trial's team revision can come from the member's checkout. observeLaunch reads the pull record of the config it is given (agent-observation.ts:101-124), and the local driver passes the run's config (local-driver.ts:603-612). Criterion 3 cannot tell, because a push does not change the member's record. Extend it: the member checkout pulls a newer team commit during the run, and every trial still records the frozen baseline.
Step 3's frozen model cannot go through chooseStepModel. It always calls resolveModelValue (step-model.ts:93-109), which returns a literal without effort (aliases.ts:521). The local driver should skip it for a trial and pass model and effort as the lease gives them.
Scorers on a trial. Say that a suite's scorers become the scorers: of each trial's prompt step, not its check step, and the generated correctness scorer is added there too. Per the note on [feat] Add workflow scorers and recorded quality labels #1085, a trial's scorer goes to the driver that holds the trial's clone and does not itself continue the copy; the clone stays until its evidence is prepared.
Give the generated correctness scorer its own identity. The record's scorers[] needs a name and a rubric hash for it, and it has no file. Give it an identity no team scorer file can take, with a rubric hash per task over the generated prompt and the task's successCriteria.
Labels stay out of the scorer view. Benchmark labels belong to this table, not to [feat] Show scorer results in the dashboard #1182's pass rates of everyday work; the exclusion is proposed on [feat] Show scorer results in the dashboard #1182. If this issue lands after [feat] Show scorer results in the dashboard #1182, it adds the exclusion there.
Regression tasks from fixed failures. [feat] Open improvement PRs proven by a before/after benchmark #1249 turns each failure an improvement PR addresses into a task of a team regression suite and runs it with [feat] Compare team asset sets in benchmarks #1181 arms. That only needs this issue's task format and the recorded start commit; nothing here changes for it.

Observing a trial through its own config and the optional ref build on F2 and F6 in #1246, which this issue now waits for.

Suites should be placed by namespace the way workflows, prompts and scorers are, so a role or project gets only its own suites:

benchmarks/<suite>/suite.yaml          # shared, every member
benchmarks/<ns>/<suite>/suite.yaml     # only members with <ns> active

# manifest/roles.yaml
roles:
  - id: auth
    resources:
      benchmarks: [auth]

Pull caches only the suites of active namespaces, plus shared ones. A benchmarks/<dir>/ that no role or project declares stays shared, the same rule as prompts/ and scorers/.
teamai push --role / --project places a suite like the other namespaced types (feat(push): support all namespaced resource types #1234).
review --branch checks every namespace's suites, not only the active ones.
A suite's scorers and model aliases resolve the same way a namespaced workflow's do.

The PR for this (#1379) follows the issue and the notes above. Where it departs from the text:

teamai prepares the trial, not the driver. A driver asks with teamai workflow next <run> --prepare. teamai makes the fresh clone with no remotes, installs the pinned team revision in it, runs setup, and answers a lease with cwd, trial.tool, trial.model, trial.effort and envScope.withhold. The driver only launches that tool. On report, teamai commits the work and collects the evidence the judges read. The next next --prepare restores protect, runs the check in the same clone and removes it. So the skill driver, Paseo or any custom driver gets the same isolation without reimplementing init, the pinned pull or the restore, and none of them depends on the local driver. A trial is leased only to a driver that sends --prepare, declares can.isolation: sandbox and lists the tool in can.trials.
The sandbox can be declared per process. workflow start and resume take --sandbox, next to workflows.sandbox: true in config. A CI job declares it on the command line, so the notice reads ...and declare it with --sandbox. instead of declare it in the config.
Aliases are the team's unless the member asks. Arm models resolve with the team's models/aliases.yaml at the pulled revision. --member-aliases lets the member's own file win, and the table shows member as the source of each model it decided.
Exits 129 to 192 count as a check that could not run outside Windows, since a shell reports a signal kill that way. The guide says not to use them as a check's own failure code.
An arm value that is not a plain name refuses the start. A tool, model or effort that looks like a URL or a path would reach the published record, so start refuses it with arm_unpublishable.
Start says how to fix a runner with no AI tool. When init ran without --agent, the issue's message (push and pull again) does not help, so start asks for --agent <tool> instead, with the same code.
The table has two more parts. After the per-arm table come a usage table (median tokens, median duration, cost when the tool stated one) and a task by arm pass-rate matrix, so each task's result is visible. The not-run list and caption follow.

Left out because they are a different problem: teamai roles set exiting 0 without a manifest, and uninstalling workspaces from the old layout, which has no registry to list them.

FYI, nothing to change here: #1389 adds session-to-task, a default workflow that writes a task in this issue's format from an archived session, and #1385 moves archived sessions to the orphan branch teamai-sessions, which replay would read.

Correction to my earlier comment: session-to-task is now a slice of #1388 that waits for this issue's task format, and archived sessions go to teamai-reports (#1385), not a new branch.

The description before the change to teamai benchmark and the default teamai-benchmark workflow is preserved below.

Original description

Parent: #1137

Goal

Labels from sessions and workflow runs show how agents did on whatever work came along. They cannot say whether a different tool or model would do better on the same work, because the work differs from run to run. A benchmark answers that: the same tasks, from the same starting commit, judged by the same check, run once per arm and repeated.

This issue adds suites with arms that change the tool or the model. Arms that change the team's skills, rules or instructions are #1181.

Files

In the team repo, next to workflows/:

# benchmarks/auth-fixes/suite.yaml
name: auth-fixes
description: Bugs from the auth module, fixed by the implementer role
repo: https://github.com/acme/api          # the project the tasks start from
defaults: { tool: claude, model: strong }  # model is optional; use an alias when arms change the tool
arms:
  - name: baseline
  - name: codex
    tool: codex
  - name: fast
    model: fast
repetitions: 3                             # 1 to 10, default 3
timeout: 6h                                # required, the run's timeout (#1082)
setup: npm ci                              # optional, runs in each trial workspace before the agent (#1082)
scorers: [diff-quality]                    # optional, scorers run on every trial (#1085)

# benchmarks/auth-fixes/tasks/expired-token.yaml
title: Reject expired refresh tokens
prompt: |
  Refresh tokens past their expiry are still accepted. Fix it.
startingCommit: 3f2c9a7d1e0b4c6a8f5d2e1b0c9a8f7e6d5c4b3a   # full 40-character commit of repo
check:
  run: npx vitest run test/auth/refresh.test.ts
  protect: ['test/**', 'vitest.config.ts', 'package.json', 'package-lock.json']
source: { platform: claude-code, sessionId: 4b1e... }   # optional, the session the task came from

A task has exactly one of check and successCriteria. successCriteria is text a correctness judge reads, for work no command can decide.

Validation, at pull, reported per file like workflows:

1 to 6 arms with unique names. Each arm changes at most one of the defaults keys, and at most one arm changes none.
A model name in defaults.model while arms change the tool is a warning: an alias resolves per tool, a name does not.
startingCommit is a full commit. check.run is non-empty, and check.protect is a non-empty list of paths relative to the repo. A task without check has successCriteria.
timeout is set. When it passes, the next call to the run settles the trials no driver took as not_run ([feat] Enforce workflow requirements and execution limits #1082). Cancelling them, once [feat] Add manual run controls and human gates #1081 adds cancel, settles them sooner.
A suite with no valid task is not cached. The last valid version is kept, as for workflows.
teamai review --branch reports the same diagnostics as fail findings, as it does for workflows ([feat] Validate workflows, prompts and scorers in review --branch #1248). Today it does not read benchmarks/ from the branch, so this adds it.

Running

teamai workflow start <suite> --benchmark, an option on the existing command, with the usual --driver. It creates an ordinary run, worked through next and report like a workflow. --driver local runs it in process. The skill driver (#1086), the Paseo adapter (#1087) or a custom driver leases its trials, checks and scorer work once it follows the trial rules below. Nothing below is specific to the local driver unless it says so.

The plan is frozen at start. Before any trial is leased, the run stores in its state, the way startRun (engine.ts) already stores the definition and sourceHash: the sourceHash of the suite file and of each task file, each task's starting commit, the ordered trial list, each arm's intended tool and resolved model and effort, the baseline team revision, the role and projects of the config the run starts in, and the rubric.sourceHash of every scorer used. A later pull does not change a running benchmark. The baseline team revision is the one #1176 reads for the checkout the run starts in. When #1176 would leave team out or report team.localChanges, or that commit is not on the team repo's remote, the run fails at start, before any trial: Benchmarks start from the team revision this checkout last pulled, and it has none that another machine can pull. Push any team changes, run teamai pull, and start again. An arm whose model alias gives no model for its tool, under #1177's rules, fails the run at start, before any trial: Arm "codex" uses model alias "strong", which has no model for codex in models/aliases.yaml. Add one and start again.

Trials. One per task × arm × repetition, ordered by repetition, then task in file name order, then arm. In repetition r, counting from 0, the arms are rotated by r (A B C, then B C A, then C A B), so no arm always runs first. The order is not a dependency: a failed or unscored trial never skips another.

Each trial:

A new workspace at startingCommit ([feat] Add workflow workspaces, branches, and session continuation #1080). For a benchmark it is a fresh clone of the project with its own git data and no remotes, not a worktree that shares the member's .git. Its agent configuration is the starting commit's, as [feat] Add workflow workspaces, branches, and session continuation #1080 makes it for every step. The driver then sets teamai up in it with teamai init <team repo> --scope workspace --agent <tool>, passing in --role and --project the role and projects the lease carries from the frozen plan. It then runs teamai pull --team-ref <teamRevision> ([feat] Pin a workspace to a team ref with teamai pull --team-ref #1183), with the frozen baseline team revision the lease carries in teamRevision. Every trial thus gets the same team assets on every driver, even when the team pushes during the run. When init or the pull exits non-zero, the trial ends as error. When the trial's steps are done, the driver removes its workspace and the teamai data init created for it. A trial pushes no branch. Its check continues its copy ([feat] Add workflow workspaces, branches, and session continuation #1080 continue). Its scorers are scorer work outside run.steps ([feat] Add workflow scorers and recorded quality labels #1085), leased to the driver that holds the copy, and they do not continue it. The check and the scorers therefore run on the driver that ran the prompt, and the copy stays until their evidence is prepared.
setup runs in it ([feat] Enforce workflow requirements and execution limits #1082). A failed setup ends the trial as error.
The prompt step runs the task's prompt with the arm's tool and model. The lease names the tool in tool. It carries in model and effort the model name and effort the frozen plan resolved for the arm, not an alias, so every driver runs the same model even if aliases change during the run. The driver passes them to the tool without alias lookup, as it passes a model name in a step model ([feat] Let a workflow choose the model of its prompt steps #1177). The local driver launches the lease's tool instead of its --tools mapping. A driver that runs another tool anyway shows up as a deviation. These generated step fields and their lease handling belong to this issue.
For a task with check, the check is an ordinary command step. Its generated command uses git, with hooks turned off, to restore the check.protect paths to their content at startingCommit and to delete files the agent added under them, commits included. It then runs check.run. Every driver that runs command steps applies it the same way. The command comes from the task file. Exit code 0 passes.

Each check is tried first. Before a task's first trial, setup and then the check run once in a clean workspace at startingCommit. If the check passes there, the task is reported invalid: check passes without changes and its trials do not run. If setup fails, or the check cannot start, times out or is killed by a signal, the task is reported check could not run and its trials are not_run: that is not the failure the task expects, and not proof the check works. The author lists in protect everything the check reads and the agent must not supply: tests, test config, scripts, helpers. A check whose inputs cannot be listed belongs in successCriteria.

Correctness judge. A task with successCriteria is judged by a correctness scorer generated from it: labels correct (score 1) and incorrect (score 0), passingScore 1. It runs through #1085 on the trial's prompt step, reading the diff from startingCommit, commits included, and the step's session. Sampling is 100 for every scorer on a trial.

Isolation. A trial must not be able to publish anything: no push to a real remote, no forge login, no git credential helper, no SSH agent, no authenticated MCP server or plugin of the member's. The driver that runs the trial provides the clone without remotes and removes GITHUB_TOKEN, GH_TOKEN, GITLAB_TOKEN, GL_TOKEN, CI_JOB_TOKEN, SSH_AUTH_SOCK, GIT_ASKPASS and SSH_ASKPASS from the tool's environment. The local driver does both. The rest, logins stored in the member's home and their MCP servers, only the environment can guarantee. So a trial requires isolation: sandbox, the level proposed in #1082 for containment beyond a worktree. The local driver declares it only when the member's config says the machine is one (workflows.sandbox: true, or the key #1082 added for it), such as a CI runner or a container without personal logins. If #1082 shipped without a sandbox level, this issue adds the level and that key. This strict rule applies to benchmark trials only and does not change how ordinary workflow steps treat isolation.

Trials that cannot run. A trial is leased only to a driver that declares sandbox and lists the trial's tool in can.trials. Listing can.trials commits the driver to the trial rules above: the lease's tool, model and effort used as given, the lease's team revision pulled with --team-ref, a fresh clone without remotes, the variables above removed from the tool's environment, and no branch pushed. A driver written before this issue lists no can.trials, so it never gets a trial. No driver is handed a trial it would have to release, since next hands a released step straight back to it. The local driver lists the tools its adapter runs. A driver lists can.trials only on a machine where the team does not also reach a trial through user scope, since #1183's pinned pull fails there. The local driver checks this at start and settles the trials as not_run with #1183's user-scope message. A trial no driver takes stays pending. The local driver is the run's only driver, so it settles the trials it cannot take as not_run at once and the run completes. Without a sandbox it prints This machine is not declared a sandbox, so benchmark trials did not run. Run benchmarks in CI or a container, and declare it in the config. For a tool it cannot run it prints Arm "<arm>" uses <tool>, which the local driver cannot run, so its trials did not run. With other drivers, the trials still pending settle as not_run when the suite's timeout passes, when they are cancelled once #1081 adds cancel, or when the run hits its limits. Finishing the run waits until every trial's scorer entries have settled, labelled or not (#1085). A trial without check whose correctness judge gave no label is unscored. Ordinary workflow runs settle as before.

The member's own checkout is never changed.

Results

Each trial ends as one of:

Result	When
`passed`	The check exits 0, or the correctness label passed
`failed`	The check ran to the end and failed, or the correctness label did not pass
`error`	Preparing the workspace (`teamai init`, the pinned pull), setup or the prompt step failed (tool error, timeout, budget), or the check could not start or finish (missing runtime, timeout, signal)
`unscored`	No check, and the correctness judge gave no label
`not_run`	The trial was never leased: no sandbox, no driver for its tool, the run timed out or was cancelled first, or an invalid task

The record. The end record gains benchmark: { suite: { sourceHash }, repetitions, baselineTeamRevision, scorers: [{ name, rubric: { sourceHash } }], tasks: [{ id, startingCommit, sourceHash }], arms: [{ name, tool, model?, effort? }], trials: [{ task, arm, repetition, order, step, result, durationMs? }] }. baselineTeamRevision is the full commit. It and the scorer hashes come from the frozen plan, even when no trial ran or no scorer gave a label. No prompts, commands, URLs with credentials or local paths. Each trial's step entry carries its agent ([feat] Record the agent each workflow step ran as #1176). A reader rebuilds the comparison from the record alone, not from the current suite files.
durationMs is measured around the prompt step, after setup and before any judge.
A deviation is a trial whose recorded agent contradicts its arm on a value that can be compared: the tool name, the team revision, or a concrete model id requested and a different one reported. A missing value is not verified, not a deviation. A tool's own model selector such as opus is not compared with a provider's model id.
The table. When the run ends, the local driver prints a table per arm, and every done envelope that next returns for the run carries it, so the skill driver or any other driver can show it. A run ends only inside next, never in report (docs/dev/workflow-engine.md). Per arm: intended tool and model, planned trials, passed, failed, error, unscored, not run, deviations, not verified, pass rate, median tokens and median duration. The pass rate is passed / planned trials. With no planned trials there is no rate. Medians leave out unknown values and show how many they used. Token medians name the components they add up.
Under the table: These results hold for these tasks, this run and this machine.

Task fix, `check.add` and `--try`

# benchmarks/auth-fixes/tasks/startup.yaml
check:
  run: npx vitest run test/startup.test.ts
  protect: ['test/**']
  add: [test/startup.test.ts]                # written from source.fix; the agent never sees it
source: { platform: claude-code, sessionId: 4b1e..., fix: 9a1d4e7c2b8f0a3d6e5c1b9f8a7d2e4c6b0a1f3e }

$ teamai workflow start auth-fixes --benchmark --try ../team/benchmarks/auth-fixes/tasks/startup.yaml --driver local --sandbox

source.fix is an optional full 40-character commit in the task's source: the commit that fixed the task. The schema is .strict(), so it ships with the first release.

check.add:

check.add is a list of repo files. Each is one path, with no glob, and must be matched by a protect entry the way git matches :(glob) pathspecs. It requires source.fix.
Content comes from source.fix, whether or not the starting commit already has the file. A fix that adds a case to an existing test file is the common case. Without this, restoreProtected reverts that file and the task can never pass --try.
The bytes are read from source.fix in the first of the trial's sources that has it, into an index of its own, as readStarting/restoreProtected read startingCommit, and never from the clone. freshClone fetches only startingCommit and its ancestors, and fetching the fix into the clone would let the agent read the hidden test from git.
The trial check runs restoreProtected, then writes the add paths from that index into the clone's worktree, then verifies them byte for byte the way the protected paths are verified.
The baseline writes them the same way, after freshClone and before setup.
unreachableTask (start.ts) also covers source.fix. Otherwise a run finishes every agent trial and only then fails at the check.
CheckTermsSchema and BaselineTermsSchema (terms.ts) carry add and fix.
An add path that is not a file at fix fails the check with trial_prepare_failed, naming the path and the commit.
Pull and review --branch refuse three things: a glob, a path no protect entry matches, and add without source.fix.

teamai workflow start <suite> --benchmark --try <task file>:

Reads the task file and the suite.yaml two directories up from it as written, validates both as pull does, and needs no teamai init. <suite> must be that suite's name.
Runs in a checkout of the suite's repo. It makes sure startingCommit and source.fix are reachable the way a real run does: the checkout, else a fetch by SHA from the suite's repo.
An add path missing at fix exits non-zero before setup.
At startingCommit, it writes the add files and the check must fail by its own exit.
At fix, it runs setup and restoreProtected, writes the files again, and the check must pass.
A setup failure or a check that could not run, at either commit, exits non-zero naming the commit and the cause. successCriteria validates the files and exits 0: there is no check to try. No arms, trials, judges or record. Needs --sandbox.

The two --try messages, verbatim: "check passes without changes at " and "check fails at the fix ".

task          at startingCommit   at fix           result
pod-startup   fails ✔             passes ✔         passes (test hidden in add)
changed-test  fails ✔             passes ✔         passes (case added to existing file, listed in add)
not-added     fails ✔ (no file)   fails ✖          "check fails at the fix <fix>"
add-missing   not tried           not tried        "<path> is not a file at the fix <fix>"

Acceptance criteria

Pull reports a suite with 7 arms, a suite without timeout, an arm that changes both tool and model, a task with both check and successCriteria, a task with an empty protect, and a startingCommit that is not a full commit, each with its file.
In a declared sandbox, a fixture suite with 2 tasks, 2 arms and 2 repetitions runs 8 trials. The record lists all 8 with task, arm, repetition and order, and the arm order of repetition 2 is the reverse of repetition 1. When the first trial fails, the other 7 still run.
Every trial's agent.project.commit is the task's startingCommit, its workspace has no remotes and its own git data, and git status of the member's checkout is the same before and after the run. Every trial's agent.team.revision is the frozen baseline team revision, also when a team commit is pushed during the run. A checkout with edited team assets, or one whose team revision is not on the remote, fails the run at start with the message above.
Inside a trial, env shows none of the variables listed above.
A trial that edits vitest.config.ts to skip the test, or adds a helper under test/, is checked against the restored files and fails.
A task whose check passes on startingCommit is reported invalid and none of its trials run. A baseline whose setup fails, or whose check cannot start, times out or is killed, is reported check could not run, and its trials are not_run. A fixture shows the failing baseline passes with a known correct patch. A check that times out after a successful trial prompt gives error, not failed.
In a model arm, each trial's agent.model.requested is the model the frozen plan resolved for the arm, on every driver, also when models/aliases.yaml changes during the run. In a tool arm, each trial's agent.tool.name is the arm's tool. An arm whose alias has no model for its tool fails the run at start with the message above, and no trial runs.
A trial whose reported model contradicts a concrete requested id counts as a deviation and stays in the denominator. A trial with no reported model is not verified.
A successCriteria task has a correctness label on each trial. A trial whose judge fails is unscored, not passed.
On a machine not declared a sandbox, every trial is not_run, the sandbox message above is printed, and the run completes. In a declared sandbox, an arm whose tool the local driver cannot run has its trials not_run with the Arm "<arm>" uses <tool> message, and the other arms' trials run. One passed trial out of ten planned shows a pass rate of 10%.
Editing the suite and pulling during a run does not change that run's plan. After removing the suite from the cache, the printed numbers can still be computed from the published record with the jq command the docs give. The record keeps the baseline team revision and every scorer's rubric hash even when observations are missing, a scorer failed, or every trial is not_run.
A fake external driver that calls teamai workflow next and report, declares sandbox and lists only claude in can.trials runs a one-task suite with repetitions: 1, a Claude arm and a Codex arm. It runs the Claude trial and its check, is never leased the Codex trial, and the suite's timeout settles the Codex trial as not_run. The done envelope that next then returns carries the table. A driver that does not declare sandbox, or lists no can.trials, gets no trial lease.
A branch holding the 7-arm suite from criterion 1 fails teamai review --branch with exit 1 and a fail finding for that file.

Test plan

Parser tests for criterion 1.
Engine tests for the frozen plan, the trial list, the rotation, independent trials, the invalid-task paths, not_run settlement and the results table, with a fake tool.
E2E with a fixture project repo and a fake tool for criteria 2 to 6 and 10 to 12, with the sandbox declared in the test config except for the first case of criterion 10. In criterion 12 the fake driver declares it in its next payload.
One real-CLI run for the PR body: a one-task suite with two arms on Claude in a container or CI job, plus Codex if available.

Watch out for

[feat] Add workflow workspaces, branches, and session continuation #1080 runs each tool "with the agent configuration from the run's base". A trial adds on top of it the team assets its pinned pull delivers ([feat] Pin a workspace to a team ref with teamai pull --team-ref #1183), and nothing resets them before the prompt step. Make that the only exception.
--benchmark is new on workflow start. Regenerate skill-data/core/references/commands.md with npx vitest run commands-reference -u.
New guide pages: docs/guide/benchmarks.md and docs/guide/zh-CN/benchmarks.md. They must say what protect has to cover, why trials need a sandbox, and to keep the suite's timeout below the host's job limit, since a timeout takes effect only on the next call to the run.
The README agent matrix (from docs: generate the tool support page from the descriptor (#1221) #1406): add benchmarks to the Team Improvement bullet and a benchmarks column in every README*.md, ✓ for the tools a trial runs on (the local driver's LAUNCH_TOOLS) and — for the rest. readme-matrix.test.ts checks the column against that list. docs/product-overview.md and its Chinese page link the benchmark guide from the Team Improvement layer.
can.trials is a new driver capability, checked before leasing like the ones [feat] Enforce workflow requirements and execution limits #1082 checks. The skill driver ([feat] Add the session skill driver and invocable workflows #1086) and the Paseo adapter ([feat] Add a Paseo driver adapter #1087) send it only once they follow the trial rules.
The guide pages carry a CI job to copy, for GitHub Actions and for GitLab CI: install teamai and Claude Code, log the tool in from a CI secret, declare the sandbox in the job's config, and run teamai workflow start <suite> --benchmark --driver local. The PR's real-CLI run uses that job.
A suite's defaults sets its baseline arm. It is not a workflow's defaults ([feat] Let a workflow choose the model of its prompt steps #1177), and the guide pages do not present them as one key.

Out of scope

Arms that change team assets ([feat] Compare team asset sets in benchmarks #1181).
Turning a past session into a task ([feat] session-to-task: turn a session's bug fix into a benchmark task #1402).
A results page ([feat] Show scorer results in the dashboard #1182 covers labels), uncertainty intervals and a recommendation.

Dependencies

Blocked by: [feat] Add workflow workspaces, branches, and session continuation #1080 (a workspace per trial), [feat] Enforce workflow requirements and execution limits #1082 (isolation levels, setup, timeouts), [feat] Add workflow scorers and recorded quality labels #1085 (scorers on trial steps), [feat] Record the agent each workflow step ran as #1176 (what each trial ran as), [feat] Let a workflow choose the model of its prompt steps #1177 (the model of an arm), [feat] Pin a workspace to a team ref with teamai pull --team-ref #1183 (the team revision of a trial), [feat] Close the #1080 gaps that #1175 leaves open #1246 (a trial's own base and optional ref, F2 and F6)
Unblocks: [feat] Compare team asset sets in benchmarks #1181, [feat] Score what a workflow run delivered #1247

docs: add the benchmarks column to the README matrix, and a changelog…

feat(scorers): show the session judge the team skills it had and used (…

feat(workflow): run benchmark suites as a default workflow, and keep …

Navigation Menu

Repository navigation

[feat] Run benchmark suites as workflows #1180

Goal

Files

Running

The default workflow

A team's own `teamai-benchmark` workflow

What each trial gets

Results

Task fix, `check.add` and `try`

Acceptance criteria

Definition of done

Test plan

Watch out for

Out of scope

Dependencies

Goal

Files

Running

Results

Task fix, `check.add` and `--try`

Acceptance criteria

Test plan

Watch out for

Out of scope

Dependencies

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[feat] Run benchmark suites as workflows #1180

Description

Goal

Files

Running

The default workflow

A team's own teamai-benchmark workflow

What each trial gets

Results

Task fix, check.add and try

Acceptance criteria

Definition of done

Test plan

Watch out for

Out of scope

Dependencies

Activity

SaulMoro commented on Oct 20, 2026

SaulMoro commented on Oct 23, 2026

SaulMoro commented on Oct 25, 2026

SaulMoro commented on Oct 25, 2026

SaulMoro commented on Oct 25, 2026

SaulMoro commented on Oct 30, 2026

Goal

Files

Running

Results

Task fix, check.add and --try

Acceptance criteria

Test plan

Watch out for

Out of scope

Dependencies

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions

A team's own `teamai-benchmark` workflow

Task fix, `check.add` and `try`

Task fix, `check.add` and `--try`