[Proposal] Score agent sessions and run benchmarks as workflows

Superseded. The revised proposal is in this comment, and its work is split into the issues listed there. The body below is kept for the discussion that led to it.

Related: #818 (workflow runtime), #1085 (scorers on workflow steps), #1083 (run records), #408 (asset health: execution effect, skill lift), #1135 (session read side). Earlier attempt: #200.

Goal

Show whether a team's skills, rules, agents and model choices make agent work better. Use two inputs: sessions members already ran, and benchmark runs that can be repeated. Add no server and no new command, and keep transcripts on the machine that produced them.

Warp Factories ships scorers and Factory Benchmarks for the same goal. This proposal keeps their scorer format, so a team can bring its scorers over, and closes the gaps listed at the end.

Scoring

One scorer definition, one sampling rule and one label record for every source.

flowchart LR
    subgraph sources [what gets scored]
        step["workflow step<br/>succeeded agent step"]:::wf
        trial["benchmark trial<br/>a workflow step too"]:::wf
        sess["finished session<br/>any tool, member opts in"]:::sess
    end
    subgraph machine [member's machine]
        sample{"deterministic sample<br/>hash of scorer + subject"}:::teamai
        read["AgentSession.find + transcript<br/>redacted, plus the diff"]:::teamai
        judge["scorer agent<br/>member's tool and model"]:::teamai
        reason[("judge reasoning<br/>stays local")]:::local
    end
    subgraph reports [teamai-reports branch]
        run[("run record<br/>label per step")]:::git
        scores[("scores/&lt;member&gt;/&lt;month&gt;.jsonl")]:::git
        stats[("stats per scorer, agent,<br/>model and source")]:::git
    end
    dash["dashboard #408"]:::ui
    step & trial --> sample
    sess --> sample
    sample --> read --> judge
    judge --> reason
    judge -- "source: workflow" --> run
    judge -- "source: session" --> scores
    run & scores --> stats --> dash
    classDef wf fill:#2f6f9f,stroke:#1f4f73,color:#fff;
    classDef sess fill:#6a4f8f,stroke:#513b6d,color:#fff;
    classDef teamai fill:#2f8f4f,stroke:#1f6b39,color:#fff;
    classDef git fill:#b5762f,stroke:#865520,color:#fff;
    classDef local fill:#555,stroke:#333,color:#fff;
    classDef ui fill:#8f2f5f,stroke:#6b1f45,color:#fff;

Scorer file. scorers/<ns>/<name>.md takes Warp's scorer.md fields: name, description, agents, labels (1 to 20 { value, score, description }), passingScore, samplingRate (0 to 100, default 25, 0 disables), and model (an alias or a model name). At least one label must be at or above passingScore and one below it. [feat] Add workflow scorers and recorded quality labels #1085 defines the file; this proposal only adds sources.
Sampling is deterministic. A subject is scored when hash(scorer, subject) mod 10000 < samplingRate × 100. Every machine picks the same sample, and a rerun of the scorer replaces its earlier label for that subject instead of adding a second one.
What a scorer reads. For a workflow step, its diff ([feat] Add workflow workspaces, branches, and session continuation #1080) and its session. For a session, the session. Sessions are read through AgentSession.find and AgentSession.transcript (feat(session): safe session reads and copies for workflows, drivers, scorers, and benchmarks (#593 follow-up) #1135), always redacted, with tool results kept apart from what the member and the agent wrote.
Where it runs. On the member's machine, with the member's tool and model, like the workflow driver. Transcripts never leave the machine.
Sessions outside workflows are opt-in. A member turns session scoring on in their local config. It is off by default, as auto_start is in Discussion: Workflow Runtime Integration #818: no tokens are spent without the member.

Label record

This is the decision to make now in #1085, so that a session label later needs no migration:

type ScoreLabel = {
  scorer: string;          // scorer name
  label: string;           // one of the scorer's label values
  score: number;           // 0 to 1, from that label
  passed: boolean;         // score >= passingScore when it was scored
  source: 'workflow' | 'session';
  subject:
    | { runId: string; step: string }            // step path, e.g. ship.check.classify
    | { platform: string; sessionId: string };   // native session id
  agent?: string;          // team agent, when there is one
  tool: string;
  model?: string;
  modelAlias?: string;
  teamRevision: string;    // team repo commit the scorer was read from
  scoredAt: string;
};

Workflow labels go in the run record ([feat] Publish privacy-preserving workflow run records and stats #1083), next to the step they judge.
Session labels go in scores/<member>/<YYYY-MM>.jsonl on teamai-reports, one line per label. Each member writes only their own file, the same split votes/ uses, so two machines never conflict. Readers keep the newest line per (scorer, subject).
Stats sum labels per scorer, agent, model and source ([feat] Publish privacy-preserving workflow run records and stats #1083).
Never published: prompts, transcripts, diffs, and the judge's reasoning. The reasoning stays in local state (the run state for a workflow, ~/.teamai/ for a session), so the member can still see why a run failed.
passed is stored, not derived. A later change to passingScore would otherwise rewrite old pass rates. teamRevision shows which version of the rubric gave each label.

Benchmarks

A benchmark is a workflow, not a command. Each trial is an agent step, so it gets a workspace, a run record and labels with no new mechanism.

flowchart LR
    past["past session<br/>clean start commit recorded"]:::sess
    task["benchmarks/&lt;suite&gt;/tasks/&lt;task&gt;.yaml<br/>prompt, start commit, check, criteria"]:::git
    wf["benchmark workflow<br/>for_each task x arm x repetition"]:::teamai
    subgraph arms [arms: one variable each]
        a1["A: claude, opus, team skills"]:::wf
        a2["B: codex, gpt, team skills"]:::wf
        a3["C: claude, opus, skill X removed"]:::wf
    end
    ws["fresh workspace at the start commit<br/>#1080"]:::teamai
    check["check command<br/>exit code"]:::teamai
    corr["correctness scorer<br/>only where no check decides"]:::teamai
    report["comparison per arm<br/>pass rate, cost, tokens, time"]:::ui
    past -- "becomes a task" --> task --> wf --> a1 & a2 & a3 --> ws
    ws --> check & corr
    check & corr -- "labels, source: workflow" --> report
    classDef wf fill:#2f6f9f,stroke:#1f4f73,color:#fff;
    classDef sess fill:#6a4f8f,stroke:#513b6d,color:#fff;
    classDef teamai fill:#2f8f4f,stroke:#1f6b39,color:#fff;
    classDef git fill:#b5762f,stroke:#865520,color:#fff;
    classDef ui fill:#8f2f5f,stroke:#6b1f45,color:#fff;

Suite. benchmarks/<suite>/suite.yaml takes name, description and arms (1 to 6). Each arm changes one thing: model, tool, or the set of team assets (a skill, rule or instruction file removed or replaced). repetitions defaults to 3.
Task. benchmarks/<suite>/tasks/<task>.yaml takes title, prompt, startingRef (repo and full 40-character commit), check (a command whose exit code decides), successCriteria (for the correctness scorer where no check decides) and an optional source ({ platform, sessionId }).
From a session to a task. A session started on a clean tree already records its start commit (feat(session): safe session reads and copies for workflows, drivers, scorers, and benchmarks (#593 follow-up) #1135), so a member can turn a real session into a task. A session that started with tracked changes is refused, because its start state cannot be rebuilt. [Proposal] rules/skills 效果评测 harness + 社区可贡献的评测集 #200 had to recover that commit from tool output.
Running it. A shipped workflow runs for_each over task × arm × repetition. Each trial starts in a fresh workspace at startingRef and runs through whatever driver the member uses. The report compares the arms on pass rate, cost, tokens and time, task by task.

Compared with Warp Factories

	Warp Factories	This proposal
Scorer file	`scorer.md`	The same fields, so a Warp scorer can be reused
Sampling	A percentage of runs	Deterministic: the same sample on every machine, and reruns are idempotent
What is scored	Runs in Warp's cloud	Workflow steps, plus any local Claude Code, Codex, OpenCode, Cursor, CodeBuddy or WorkBuddy session
Where traces live	Warp's cloud or the company's storage	Only on the member's machine. Only the label is published.
Rubric version	Not recorded on a score	`teamRevision` on every label
Benchmark grading	LLM correctness scorer	The `check` exit code first, a scorer only where no check decides
Benchmark arms	`model` or `harness`	`model`, `tool`, or the set of team assets, which gives the effect #408 calls skill lift
Benchmark source	A past Warp run	A past session from any supported tool, with its start commit
Runner	Warp's cloud runners	A workflow run by any driver

Out of scope

Dashboard pages (#408), automatic improvement PRs from failing labels (selfImprovement), scoring subagents at SubagentStop, scorer access to secrets, and new CLI commands.

Open questions

How are sessions picked for a scorer outside workflows? Warp scopes a scorer to agents. A session in a project has no team agent. One option is agents: [{ session: true, tools: [...] }].
When is a session finished? Proposal: idle for 6 hours, scored by the background pull at the next session start, with a daily budget per member.
Should a benchmark suite live next to the skill it tests (as Proposal: Team Context Assets Health — Skills / Rules / CLAUDE.md 多维质量与效果评估 #408's evals/ proposes), or in its own benchmarks/ namespace?

Two things from #1148 that the benchmark workflow runs into:

Expressions cannot build task × arm × repetition. for_each takes one list, and expressions have length, any, all and filter, with no map, product or range. An item of a nested for_each in an inline workflow reads only its own value, not the outer item. The simplest trial list comes from a command step that prints it as an output.
The local driver runs agent steps one at a time. It starts a prompt step only when nothing else runs, because the read-only check compares the whole worktree before and after the prompt. On --driver local, trials run in sequence whatever max_parallel says, until [feat] Add workflow workspaces, branches, and session continuation #1080 gives each step its own worktree. At 3 arms, 3 repetitions and 20 tasks, that is 180 agent runs back to back.

Revised proposal

What changed from the body above:

"Agent" means what ran, not a file. In labels, records and benchmarks, agent is the composition the model worked inside, recorded as observed.
Workflows do not wait on this. [feat] Add workflow scorers and recorded quality labels #1085 defines the scorer file, the scorers: step key and the label that sessions reuse. Everything here builds on workflow issues. The only workflow issue that waits on these is [feat] Open improvement PRs proven by a before/after benchmark #1249 (improvement PRs), on [feat] Compare team asset sets in benchmarks #1181, [feat] Show scorer results in the dashboard #1182 and [feat] Compare scorers with hand labels, including on pushed sessions #1269.
Benchmarks change tool, model or team assets, judged by a trusted check, in a sandbox. The plan is frozen at start, arms alternate in order, and every planned trial stays in the denominator.
The comparison with Warp Factories is corrected.
The open questions are closed (below), and the work is split into seven issues, plus a pull option ([feat] Pin a workspace to a team ref with teamai pull --team-ref #1183) that benchmarks use.

What "agent" means

A model never runs as a file in agents/. It runs inside a composition: the definition, if any, and how the tool applied it; the tool and its version; the model and effort, as requested and as the tool reported them; the team revision TeamAI delivered to the checkout, the roles, projects and tags it was delivered for, and whether a delivered file was edited; and the project commit. #1176 records this for every workflow pass, and #1178 for every session, when it starts. #1178 adds the roles, projects and tags to both. A field that was not observed is left out, never guessed. A fingerprint over the shared parts is written only when tool, model and team revision are all known, so "same agent" is never claimed from partial data.

This is what TeamAI can record that a hosted factory cannot: what each member's tool actually had, on every tool, without publishing transcripts to team reports. The step key agent: still selects a definition, and model: on a prompt step (#1177) picks a model without one.

Names. teamRef is a branch, tag or commit of the team repo, as a person writes it. teamRevision, and team.revision inside agent, is the full 40-character commit it resolved to. On the project side a commit is always written in full: startingCommit in a benchmark task, project.commit in agent.

Scoring

Scorer file and workflow labels: [feat] Add workflow scorers and recorded quality labels #1085. The classification and rubric fields of Warp's scorer.md, the scorers: step key, deterministic sampling, passed stored when scored. The subject names the pass, with rubric: { sourceHash, teamRevision? } and judge: { kind: "model", tool, model?, usage? } | { kind: "human" }. A hand label ([feat] Compare scorers with hand labels, including on pushed sessions #1269) has judge: { kind: "human" }.
Two homes, one label. A workflow label sits in the run record next to the pass it judges and takes agent from that pass. A session label goes to scores/<member>/<YYYY-MM>.jsonl on teamai-reports and carries agent itself. The key is (scorer, rubric.sourceHash, subject, judge.kind) in both.
Sessions on demand ([feat] Score a finished session on demand #1178). teamai score --scorer <name> [--session <id> [--source <platform>]] [--push], a command of its own rather than a session save option, since scoring shares none of save's flags and its subject is not always a session. It ignores sampling, so a team can calibrate a scorer on a few sessions before sampling more. A label is about the whole session, and carries agent only when the session ran as one agent throughout.
Sessions in the background ([feat] Score finished sessions in the background #1179). Off unless the member turns on scoring.sessions, and only for scorers with sessions: true. A session is finished after 6 hours idle. Each session is tried once per rubric, and a daily limit counts every judge launch.
The judge can only answer. Commands, file access, web, subagents, MCP and the member's hooks and plugins are turned off, and each tool's profile is verified on the real CLI. It runs in an empty directory, without forge or git credentials, on the member's tool and model. A tool whose installed version cannot be run that way is not used.
What is published: labels only. Transcripts, prompts, diffs and the judge's reasoning are not published to team reports. The judge's inference goes to the member's own model provider, as their sessions do.

Benchmarks

Files. benchmarks/<suite>/suite.yaml (repo, defaults, 1 to 6 arms, repetitions, a timeout, optional setup and scorers) and benchmarks/<suite>/tasks/<task>.yaml (prompt, starting commit, a check or successCriteria).
Running. teamai workflow start <suite> --benchmark builds a workflow run: one trial per task, arm and repetition, each a prompt step in a new workspace at the starting commit, then the check.
Arms change one thing: the tool or the model ([feat] Run benchmark suites as workflows #1180), or the team repo revision ([feat] Compare team asset sets in benchmarks #1181), which measures a skill or rule the way Proposal: Team Context Assets Health — Skills / Rules / CLAUDE.md 多维质量与效果评估 #408's skill lift describes.
A check decides first. It runs from the task file after the protected paths (tests, test config, scripts) are restored, and it must fail on the starting commit, or the task is reported invalid. A correctness judge is used only where no command decides.
Isolation. A trial must not be able to publish anything. The driver that runs a trial gives it a clone without remotes and an environment without forge or git credentials. Trials go only to drivers that declare they follow these rules, on a machine declared a sandbox, such as CI or a container, since logins stored in a member's home are out of TeamAI's reach. Trials no driver can take are reported as not run.
Reporting. The plan (tasks, starting commits, each arm's resolved tool, model and team revision, scorer rubrics) is frozen when the run starts and published with it. Arms rotate in order across repetitions. The pass rate is passed over planned trials. A trial whose recorded agent contradicts its arm is a deviation, and one that cannot be checked is reported as not verified. The result is stated for those tasks and that run, not as a general ranking.

Labels from sessions and workflow runs show how agents did on the work that came along. Benchmarks compare tools, models and team assets on the same tasks. The scorer view in the dashboard (#1182) fixes the rubric, the source and the judge before it groups by any part of the agent.

Compared with Warp Factories

	Warp Factories	This proposal
Scorer file	`scorers/<name>/scorer.md`, with runner, secret and MCP settings besides the rubric, and the agents it judges	The classification and rubric fields, in `scorers/<name>.md` (#1085). The workflow step names its scorers in `scorers:`. Porting a Warp scorer means moving its path, replacing `agents` with `scorers:` on the steps, and adapting `model`
Sampling	Deterministic per scorer and run. An omitted rate means 100% in the API and 25% in the file	Deterministic per scorer and subject, decided the same way on every machine with no server. One default, 25
What is scored	Factory agent runs, on Warp's runners or self-hosted workers, and with `includeDescendants` their child runs. Sampled automatically, or one run on demand	Workflow steps on any driver, what a run delivered (#1247), and sessions of supported tools, the member's own or pushed to the team repo (#1269). On demand with `teamai score`, or in the background if the member opts in
Where traces live	Warp. Enterprise can keep them in its own S3, which Warp still reads	The member's machine. Only labels are published
Rubric version on a result	Versioned internally, not exposed. Pass/fail is recomputed against the current threshold	`rubric.sourceHash` and, when resolvable, `teamRevision`. `passed` is kept as scored
Rescoring	Replaces the result	The same rubric replaces. An edited rubric adds a label, so rubrics can be compared
Agent recorded per run	Model, harness and reasoning level, runner, skill paths without a commit, MCP	Definition and how it was applied, tool and version, model and effort as requested and reported, team revision with the roles, projects and tags it was delivered for, local edits, project commit. For a session, the team skills it loaded
Hand labels	Not in the docs; scorers are checked by hand	A member labels a session with a scorer, and the dashboard shows how often the judge agrees (#1269)
Pass rate breakdown	By period. Model and harness only for run counts	By any recorded part of the agent, with rubric, source and judge fixed
Benchmark grading	An LLM correctness scorer, plus chosen scorers	A trusted check first, required to fail on the starting commit. A correctness judge only where no command decides
Benchmark arms	Model, harness, runner or router, per named agent	Tool, model or team repo revision. Warp has no skill or rule arm
Benchmark runner	Warp's runners	Any workflow driver that follows the trial rules, in a declared sandbox
Cost of adoption	Paid plan for third-party harnesses	No server and no plan
Self-improvement	PRs from failing runs	Not in this proposal

The open questions

Picking sessions for a scorer. A scorer says sessions: true, and the member turns on scoring.sessions. A workflow step lists its scorers in scorers: ([feat] Add workflow scorers and recorded quality labels #1085), and a benchmark suite lists them in suite.yaml.
When a session is finished. After 6 hours idle. Scored after the background pull of a later session start, within a daily limit.
Where suites live. In benchmarks/<suite>/, not next to a skill, because an arm can change several assets at once. Proposal: Team Context Assets Health — Skills / Rules / CLAUDE.md 多维质量与效果评估 #408 can present results per asset.

Issues

Arrows point from an issue to what it unblocks.

flowchart LR
    subgraph wf ["#818 workflows"]
        i1080["#1080 workspaces"]
        i1082["#1082 requirements"]
        i1085["#1085 workflow scorers"]
        n1["#1176 agent each step ran as"]
        n2["#1177 model of prompt steps"]
        i1090["#1090 PR delivery"]
        i1246["#1246 workspace gaps"]
    end
    subgraph sc ["#1137 scorers and benchmarks"]
        n3["#1178 score a session on demand"]
        n4["#1179 score sessions in the background"]
        n5["#1180 benchmark suites"]
        n6["#1181 team asset arms"]
        n7["#1182 scorer view in the dashboard"]
        n9["#1269 compare scorers with hand labels"]
        n10["#1247 score what a run delivered"]
    end
    n8["#1183 pull --team-ref"]
    i1085 --> n3
    n1 --> n3
    n3 --> n4
    n3 --> n7
    n3 --> n9
    n7 --> n9
    i1080 --> n5
    i1082 --> n5
    i1085 --> n5
    n1 --> n5
    n2 --> n5
    n5 --> n6
    n8 --> n5
    i1085 --> n10
    i1246 --> n5
    i1246 --> i1085
    i1090 --> n10
    n3 --> n10
    n5 --> n10

#1176, #1177 and #1183 have shipped. #1183 pins a workspace to a team ref, a pull feature of its own that benchmarks use to prepare trials. #1178 and #1182 follow #1085, #1269 follows both, #1180 and #1181 come next, and #1247 comes last, after #1090 too.

Out of scope

Self-improvement PRs, scoring subagents at SubagentStop, the member's own tool config in the agent record, per-asset health pages (#408), uncertainty intervals and recommendations in benchmark reports, and new CLI commands other than teamai score (#1178).

Whole-run scoring, which the 2026-10-15 decision on #1085 kept here, is now #1247: it scores what a run delivered, next to the session and step labels, which stay as they are. Improvement PRs, which this proposal leaves out, are #1249 under #818; they reuse #1180 and #1181 to prove each change before review.

Update from #1085's design: "Two homes, one label" becomes one home.

workflow label ─┐
session label  ─┴─► scores/<member>/<YYYY-MM>.jsonl   (key: scorer, rubric.sourceHash, subject, judge.kind; newest scoredAt wins)
                    Workflow Runs joins by subject: runId + step#pass

A workflow label takes the judged pass's agent from the run at read time, so it still carries no copy of it.

Navigation Menu

Repository navigation

[Proposal] Score agent sessions and run benchmarks as workflows #1137

Goal

Scoring

Label record

Benchmarks

Compared with Warp Factories

Out of scope

Open questions

Revised proposal

What "agent" means

Scoring

Benchmarks

Compared with Warp Factories

The open questions

Issues

Out of scope

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[Proposal] Score agent sessions and run benchmarks as workflows #1137

Description

Goal

Scoring

Label record

Benchmarks

Compared with Warp Factories

Out of scope

Open questions

Activity

SaulMoro commented on Oct 16, 2026

SaulMoro commented on Oct 16, 2026

Revised proposal

What "agent" means

Scoring

Benchmarks

Compared with Warp Factories

The open questions

Issues

Out of scope

SaulMoro commented on Oct 20, 2026

SaulMoro commented on Oct 22, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions