[feat] Compare team asset sets in benchmarks #1181

Parent: #1137

Goal

The question a team asks most is whether a skill, a rule or an instruction file makes the agent better. #1180 compares tools and models. This issue adds arms that change the team's assets, and arms that change how the agent works by running one of the team's workflows.

A team prepares an asset variant as a branch of the team repo, for example bench/without-release-skill, or leaves team repo paths out of the pulled revision with without. #408 calls the effect skill lift.

# benchmarks/release/suite.yaml
defaults: { tool: claude, model: strong }
arms:
  - name: with-skill                                  # the revision this checkout pulled
  - name: without-skill
    team: bench/without-release-skill                 # a branch, tag or commit of the team repo
  - name: no-release
    team: { without: [skills/release] }               # the pulled revision without these team repo paths
  - name: no-commit-rules
    team: { without: [rules/commits.md, claudemd/team.md] }
  - name: bare
    team: none                                        # no team assets at all

A matrix crosses values into one arm each, named after the values it varies:

# benchmarks/onboarding-docs/suite.yaml
defaults: { tool: claude }
matrix:
  tool: [claude, codex]
  team: [pulled, none]

arms:  claude-pulled, claude-none, codex-pulled, codex-none
pairs: team on claude, team on codex, tool on pulled, tool on none

An arm can run a team workflow instead of a single prompt:

# benchmarks/release/suite.yaml
arms:
  - name: one-shot
  - name: plan-first
    workflow: plan-then-fix

# workflows/plan-then-fix.yaml
schemaVersion: v1alpha1
name: plan-then-fix
description: Plan the change, then make it.
inputs: { task: string }                      # required: each trial passes the task's prompt
steps:
  - id: plan
    prompt: |
      Read the code and write a short plan for this task. Change no file yet.
      {{ inputs.task }}
    output: { plan: string }
  - id: fix
    needs: [plan]
    prompt: |
      {{ inputs.task }}
      Follow this plan: {{ steps.plan.outputs.plan }}

Rules

• team is one more defaults key: pulled (the default), none, a ref (a branch, tag or commit of the team repo), or { without: [<team repo path>, ...] }. A missing team is pulled, the baseline team revision [feat] Run benchmark suites as workflows #1180 freezes.

• workflow is an arm key naming a team workflow. matrix crosses it like tool, model and team. A matrix lists workflow names only, so a matrix that varies workflow has no plain-prompt arm; put a prompt arm and a workflow arm side by side under arms.

• An arms entry changes at most one defaults key. A matrix crosses several, up to 12 arms. A derived arm is named after the values the matrix varies, joined by -, such as codex-none or claude-plan-then-fix.

• A team changes only the delivered assets. When plan runs, it freezes from the member's pulled team revision the tasks, the checks, the scorers with their rubric.sourceHash, and each arm's tool and resolved model and effort, and uses them for every arm. A branch that also edits models/aliases.yaml or a scorer does not re-resolve any arm's model or change its judge.

• A revision is a combined treatment. When the branch differs in more than one asset, the report names the arm as that revision, not as the effect of one skill.

• Resolve without fetching. plan resolves each ref to one commit from the refs the team clone already has, as teamai pull --team-ref does ([feat] Pin a workspace to a team ref with teamai pull --team-ref #1183), and freezes it per arm. A ref that does not resolve fails teamai benchmark run before a run is created (arm_team_unresolved):

  Arm "without-skill" runs on team ref "bench/without-release-skill", which is no branch, tag or commit of this checkout's team repo that holds teamai.yaml. Its branches: main. Fetch it, or fix the arm's team, push it, run teamai pull, and start again.
  

• Delivery. teamai prepares each trial's clone at lease, with the arm's team revision installed under the clone's own install home. The driver installs nothing. The observed agent.team.revision, and the skills the tool loaded, are read from that install, never from the member's config.

• User scope. Assets the team delivers at user scope, or a project that inherits user scope (inheritUserScope), load in every arm, so the arms would not differ in what they claim to. plan then refuses every suite with team_ref_blocked before a run is created ([feat] Run benchmark suites as workflows #1180). A none arm is also refused (arm_team_unclean) on a machine whose user-scope install delivers to the arm's tool:

  Arm "bare" runs without team assets, and this machine's user-scope teamai install delivers team assets to claude in your home, where its trials would read them. Start the benchmark on a machine where teamai is installed only in workspace scope.
  

  The member's copies are never removed. Context the member added on their own is outside what an arm guarantees, and the docs say so.

• The member's own checkout, its pull record and its delivered files are never changed.

Arms without team repo paths

without lists paths of the team repo, as git lists them: a file or a directory, under any asset directory, namespaces included. The arm's clone installs a commit made from the pulled revision without those paths, and its trials record the pulled revision as agent.team.revision.

team: { without: [skills/dev] }                       # a whole skills namespace
team: { without: [skills/release, skills/dev/release] }   # a shared skill and the namespace version that stands in for it
team: { without: [agents/reviewer.md, docs/runbooks] }

plan runs pull's own resolvers at the pulled revision, with and without each path, for the arm's roles, projects, tags and excluded skills. Under skills/, rules/, agents/, claudemd/ and docs/, it refuses a path whose leaving out changes nothing pull delivers, with the reason. Elsewhere a path only has to exist. All refusals leave no run:

An entry inside a shared file, such as one MCP server, hook, env key or model alias, is not a path: without cannot leave it out, and such an arm still needs a branch.

Arms that run a workflow

plan pins the workflow and the workflows it runs by name at the arm's team revision, resolved through the namespaces the member receives, as any team workflow is. It refuses, before a run is created:

The step reasons are delivers a pull request, asks a person, opens a workspace other than continue, and runs a command in the member's checkout; give it workspace: continue.

Comparing arms

A pass rate per arm hides which trials moved. teamai benchmark report derives the pairs once from the trial entries of the record, and the dashboard's Compare tab derives them the same way. Two arms are paired when they differ in exactly one of team, tool, model and workflow, trial by trial on run, task and repetition.

• A pair carries the key it differs in, the arm it is measured from and the arm it is measured to, the number of paired trials, each arm's pass rate and usage over them, each task's pass rate on both arms, and the difference with its 95% interval for paired proportions (Newcombe). A trial that is error, unscored or not_run counts as not passed. For team, the difference is measured from none, then without, then a ref, then pulled; otherwise from the arm listed first. No arm is recommended.
• The pairs reach drivers in outputs.table.pairs and show on the Compare tab. The printed text has no pair table.
• Skill loads. For an arm whose team adds, removes or changes skills under skills/ against the baseline revision (a ref arm, as git compares the two revisions; a without arm, from its paths), the printed text lists each such skill, with how many of the arm's trials whose work ran loaded it, and how many of the suite's defaults arm did, as the passes record it in used. A trial with no observation counts as unknown, never as not loaded.

| Arm | Skill | Change | Loaded in this arm | Loaded in the defaults arm |
| --- | --- | --- | --- | --- |
| no-release | release | removed | 0/6 | 4/6 |
| without-skill | release | removed | 0/5, 1 unknown | 4/6 |

A cell reads loaded/observed, adds , N unknown when some trials went unseen, reads unknown (N) when none was seen, and - when the suite has no defaults arm.

#1249 reads this from the run's done outputs, or from the benchmark record.

The record

The benchmark record (#1180) carries, per arm and per trial:

arms[]:       { name, tool, model?, alias?, effort?, source: team | member,
                team: { kind: pulled, revision } | { kind: ref, ref, revision }
                    | { kind: without, paths, revision } | { kind: none },
                workflow?: { name, sourceHash } }
trials[]:     { …, agent?, used?: { <skill>: loaded | unknown } }      a skill not listed was not loaded
skillChanges?: [{ arm, defaults?, skills: [{ name, change: added | removed | changed }] }]

Pairs are not stored; they are derived from trials[] when read.

Acceptance criteria

1. In a suite whose second arm points at a branch without skill X, that arm's trials have agent.team.revision equal to the branch's commit and no skill X in the tool's skill directory of their clone, and the first arm's trials have it. An arm with team: { without: [skills/X] } has no X in its clone, and its trials record the pulled revision.
2. A branch that also changes models/aliases.yaml and a scorer file runs its arm with the same requested model and the same scorer rubric.sourceHash as the first arm.
3. A ref that does not resolve, and a without path the pulled revision lacks, each fail teamai benchmark run before a run is created, with the arm and the fix in the message, and no fetch is run.
4. A team set up at user scope, and a project with inheritUserScope, each make teamai benchmark run fail team_ref_blocked before a run is created, and no trial runs. A none arm on a machine whose user-scope install delivers to its tool fails arm_team_unclean, naming the arm.
5. After the run, including a run cancelled halfway, the member's checkout, its pull record and its delivered files are as before.
6. The printed table names each arm's team in a Team column, such as pulled 3f9c2ab, bench/without-release-skill 81d0e4c, 3f9c2ab without skills/release or none. Every arm's record entry has team as written and, except none, the commit its trials got.
7. A suite with a default Claude arm, a Codex arm and a ref arm runs the first and third with the same frozen default tool and model, and the second with Codex and its frozen model.
8. A fake external driver leased a trial of the ref arm gets a prepared clone, runs no init or pull, and the trial records the arm's commit as agent.team.revision.
9. In a suite with arms a and b that differ only in team, 2 tasks and 3 repetitions, where every trial passes except b's trial of task 1 in repetition 2, outputs.table.pairs and the Compare tab show one pair over 6 trials, a at 6/6 and b at 5/6.
10. A trial error in either arm stays paired and counts as not passed.
11. An arm whose branch removes skill X lists X as removed, with the defaults arm's load count from the local driver's used. With a fake driver that reports no used, both cells read unknown (N).
12. A matrix of tool: [claude, codex] and team: [pulled, none] makes the four arms above, and only arms that differ in one key are paired.
13. An arm with workflow: plan-then-fix runs both its steps in the trial's clone with the arm's tool and model, is checked on what they left, and records durationMs as the sum of the two turns and workflow: { name, sourceHash } on its arm.
14. teamai benchmark run refuses, before a run is created, an arm workflow without inputs.task, one with a step that delivers, asks a person, opens a workspace other than continue, or runs a command without workspace: continue, and one that two namespaces the member receives define (workflow_namespace_conflict).
15. With trial.timeout set, a workflow arm whose first turn spends the allowance has its second step skipped, its check run, and limit: timeout in its entry. With trial.max_turns, it lists max_turns in unenforced. A workflow arm whose step fails without a limit is error.
16. teamai benchmark run refuses a without arm, before a run is created and with the messages above, for: a namespace the member does not receive, a shared path an active namespace replaces, a skill the member excludes, a rule its tags leave out, and teamai.yaml. Pull and review --branch refuse a path inside another listed path. A without of a whole namespace directory the member receives runs with none of its files in the clone.
17. A workflow arm whose workflow comes from workflows/<ns>/ for a member whose role receives <ns> runs that version, and its sub-workflows from the same namespace.

Definition of done

• docs/guide/benchmarks.md and docs/guide/zh-CN/benchmarks.md: team with its without paths and namespace rules, the limit that an entry inside a shared file needs a branch, matrix and arm workflow, the user-scope rule, the pairs and skill loads.
• docs/guide/dashboard.md and its zh-CN page: the pairs on the Compare tab.
• skill-data/core/references/commands.md regenerated if teamai benchmark help changes.
• Real-CLI record in the PR: one suite with a team arm and a workflow arm on Claude, in the sandbox [feat] Run benchmark suites as workflows #1180 requires.

Test plan

• Unit tests for ref resolution, the frozen controls, the scope check, the without refusals at pull's resolvers, the pinned arm workflow and its refusals.
• Unit tests for the pairing and for the list of changed skills between two team commits.
• E2E through the built CLI with a fixture team repo with two branches, a fixture project and a fake tool for criteria 1 to 5, 8 and 11 to 17. In criterion 8 the fake driver declares the sandbox and lists the arm's tool in can.tools.

Watch out for

• Delivery into the trial's clone must not touch the git exclude block or the pull record of the member's checkout (lastPullByWorkspace, checkoutKey).

Out of scope

• Leaving out an entry inside a shared file (an MCP server, hook, env key or model alias) without a branch, and the per-asset view Proposal: Team Context Assets Health — Skills / Rules / CLAUDE.md 多维质量与效果评估 #408 proposes.
• A recommendation of an arm.

Dependencies

• Blocked by: [feat] Run benchmark suites as workflows #1180
• Unblocks: [feat] Open improvement PRs proven by a before/after benchmark #1249