[feat] Add workflow scorers and recorded quality labels

Parent: #818

Goal

Produce comparable quality labels across drivers without coupling scoring to branch delivery or a specific runtime.

How it works

# scorers/review-quality.md
---
labels:
  - { value: good, score: 1, description: Finds the real defects and nothing else }
  - { value: noisy, score: 0.5, description: Finds them, plus nits }
  - { value: missed, score: 0, description: Misses a real defect }
passingScore: 0.5
samplingRate: 25
model: strong
---
Judge the review against the diff. Answer with one label.

# workflows/review-and-fix.yaml (one step)
- id: review
  agent: reviewer
  scorers: [review-quality]          # or { names: [review-quality], batch: true }

$ teamai workflow start review-and-fix --driver local   # the run settles; scoring never holds it
$ teamai score --next --json '{"driver":{"kind":"local","machine":"m1","can":{"judge":{"isolation":"verified","tools":["claude"],"access":"read"}}}}'
{"data":{"lease":{"ref":"s5d1e07c2a9b4/a91c0d7e","subject":{"runId":"r3f9","step":"review","pass":1},
  "evidence":"<dataHome>/scores/evidence/a91c0d7e/","answerSchema":{…},
  "submit":"teamai score --lease s5d1e07c2a9b4/a91c0d7e --submit --stdin"}}}
$ teamai score --lease s5d1e07c2a9b4/a91c0d7e --json '{"driver":{…}}'     # judge here, local profile
$ teamai score --lease s5d1e07c2a9b4/a91c0d7e --submit --stdin < answer.json   # or any driver's own judge
$ teamai score --run r3f9 --step review --pass 1 --scorer review-quality [--queue]   # on demand

workflow next ──► never hands out judge work
score --next  ──► judge lease (any driver with can.judge) ──► evidence dir ──► submit ──► label store
                                                                               scores/<member>/<YYYY-MM>.jsonl

Scope

scorers/<name>.md definitions (frontmatter plus prompt; README.md ignored, other files are errors) and the scorers: step key in its short and long forms.
Deterministic sampling of succeeded passes of the steps that list a scorer.
Open scoring work is derived from the run, never stored in it: no end, break, loop, count, failure or work-token total reaches it, and the run settles without waiting.
teamai score --next | --lease | --submit | --run … --pass … --scorer … [--queue]. A judge lease settles only through --submit; workflow report rejects it.
Evidence directory per attempt, redacted as a whole: diff.patch, transcripts/<step>#<pass>.md including subagents, repo/ (tracked files at the judged commit, no symlinks). A read-only step's diff is its recorded read.base..read.commit.
Judge profiles per tool, their verification, and an isolation policy (team scoring.isolation, scorer isolation, machine scoringIsolation; strictest wins; default best-effort; teamai doctor shows it).
Labels go to one label store shared with [feat] Score a finished session on demand #1178, with a keyed outbox: { scorer, rubric: { sourceHash, teamRevision? }, label, score, passed, source: "workflow", subject: { runId, step, pass }, judge: { kind: "model", tool, model?, usage?, isolation, access, batched? } | { kind: "human" }, scoredAt }. Key (scorer, rubric.sourceHash, subject, judge.kind); the newest scoredAt wins. Reasoning stays local in <dataHome>/scores/reasoning/<labelVersion>.md, one file per verdict (labelVersion is the sha256 of the label), so a rescore never overwrites earlier reasoning.
Workflow Runs shows {scorer, label, passed} per pass and a separate scoring total.
Unknown scorers: names: fail in teamai review --branch; a warning at pull, and the step runs without that scorer.

Out of scope

PR delivery, automatic prompt rewriting, automatic improvement PRs, arbitrary judge code, scorer access to secrets, and whole-run scoring (#1247).

Dependencies

Acceptance criteria

Sampling is deterministic for a run/step/pass/scorer tuple.
workflow next never returns judge work; teamai score --next does, only to a driver whose can.judge meets the required isolation.
A run with a sampled scorer that fails or spends tokens has the same stepCounts, iterations, failure and work token total as the same run with samplingRate: 0.
A phrase only the judge writes appears in no file under the run directory and in no published record.
A label outside the scorer's list is rejected and the lease stays open.
Unavailable scorer models follow the documented policy without changing the judged result.
Labels from every driver and from [feat] Score a finished session on demand #1178 share one type and one store.
A hostile fixture transcript gets every forbidden judge action denied under each verified profile; a tool that does not honor its profile is best-effort.
No PR or provider capability is required.
A branch with a malformed scorer or an unknown scorers: name fails teamai review --branch with exit 1 and a fail finding.

Verification

Focused scorer parsing, sampling, leasing, label, failure, and redaction tests; build/typecheck/lint; one real-CLI E2E with a fixture scorer and a real judge. Bilingual docs/guide/scorers.md, with pointers from the workflow pages.

The scorer reads the diff of the step it judges, which needs the branch or workspace from #1080. Maybe add #1080 to blocked-by?

Yes. A scorer reads the diff of the step it judges, so this issue is also blocked by #1080.

Warp's scorers judge a whole agent run, with its transcript, tool calls and outputs, rather than one step. Some suggestions from comparing the two:

A scorer could judge an agent run instead of a step. The same scorer could then run later on subagents in normal sessions, at SubagentStop, as votes-judge.ts already does. Its input would be the task, the output, the transcript when the driver has one, and the diff when there is a workspace. Reports from feat(workflow): run prompt and agent steps with the local driver #1120 already carry the session id. A transcript reference next to it, read only by the driver that wrote it (a file path for local, a thread id for a T3 Code driver), would show the judge what the agent did and not only the diff.
The file could follow Warp's scorer.md: labels as a list of { value, score, description }, samplingRate from 0 to 100 with 25 as the default, between 1 and 20 labels, and at least one label at or above passingScore and one below it. Teams coming from Warp could then bring their scorers.
The judge's reasoning stays in the local run state, and only the label goes into the record, as planned.
Scorers need a model, and a prompt step without agent could use the same key. It would take what an agent's model takes, an alias (Proposal: model aliases so one agent works in every tool #830) or a model name. next runs where the driver runs, so it could resolve the alias with resolveAgentModel for the driver's tool and return { model, effort }. With no model, the tool's default applies and the driver does not pick one.

Notes that affect scorers:

Graph edges are only explicit needs. Declaration order no longer implies a dependency (needs: [] declares another root), so "the edge a scorer judges" is a needs edge.
A step's run-time status is pending | leased | running | succeeded | failed | skipped; running is a workflow step whose child steps are in progress. Only succeeded agent steps are candidates; steps inside sub-workflows are addressed by path (ship.check.classify).
An internal scorer step must stay out of the run outcome. A run fails at the end if any step stopped (stopped() is failed today), and dependents of a stopped step are skipped. A failed scorer step counted like a normal step would fail the run or skip the judged step's dependents, which breaks "scorer failure does not rewrite the judged step's outcome".
Steps are created lazily. Child steps appear in the run only when the workflow step would be the next lease; a materialized scorer step can follow the same pattern (insert after the judged step, lease through next).
Drivers report only success | failed | released. skipped and named ends are decided by the engine, not reported.
Proposal: let a scorer read the judged step's session too. A diff shows what changed, not how the agent got there (an ignored instruction, a failed tool call retried three times). A scorer could take the judged step's session as a second input, resolved from the same reference session: { from: <step> } uses in [feat] Add workflow workspaces, branches, and session continuation #1080, and always redacted (the redaction teamai session migrate --scrub applies) before it reaches the scorer prompt, since secrets stay out of scope. Records would still keep only the scorer name and label. If this belongs in a follow-up instead, I can file it.
feat(session): safe session reads and copies for workflows, drivers, scorers, and benchmarks (#593 follow-up) #1135 (draft) adds the read side a scorer would use:
- AgentSession.find returns the session redacted by default.
- AgentSession.transcript renders one text entry per message for a judge's prompt, with tool results kept apart from what the member and the agent wrote. A scorer that counts tool calls reads them from session.messages, which has the same shape for every tool. Tokens, interruptions and corrections are already in the dashboard events under the same session id.
- The session start hook records the commit the session started on and whether tracked files had changes, so a scorer can tell which code a session ran against.
- AgentSession.list lists sessions across tools.
Proposal: one label record for every source, decided here so that labels from outside workflows ([Proposal] Score agent sessions and run benchmarks as workflows #1137) fit later without a migration: { scorer, label, score, passed, source: 'workflow' | 'session', subject, agent?, tool, model?, modelAlias?, teamRevision, scoredAt }. subject is { runId, step } for a step. Workflow labels go in the run record ([feat] Publish privacy-preserving workflow run records and stats #1083). passed is stored because a later change to passingScore would otherwise rewrite old pass rates. teamRevision shows which version of the rubric gave each label. Prompts, diffs, transcripts and the judge's reasoning are never published; the reasoning stays in the run state. Sampling is hash(scorer, subject) mod 10000 < samplingRate × 100, so a rerun replaces the label instead of adding one.

The engine rules that span files are in docs/dev/workflow-engine.md, added by #1129.

A scorer here judges one succeeded agent step, not a whole run. Whole-run scoring stays in #1137.

The step diff and its redacted session are both inputs. The session is resolved like session: { from: <step> } and redacted like teamai session migrate --scrub. Prompts, diffs, transcripts, and reasoning stay local.

Use Warp's scorer.md: 1–20 labels of { value, score, description }, samplingRate 0–100 (default 25), with at least one label at or above passingScore and one below. model is an alias or a model name; omit it to use the tool default.

Lock the label record now:

{ scorer, label, score, passed, source: "workflow" | "session", subject, agent?, tool, model?, modelAlias?, teamRevision, scoredAt }

subject is { runId, step }. Sampling is hash(scorer, subject) mod 10000 < samplingRate × 100. A failed scorer does not change the judged step or the run.

Because the label record carries teamRevision, it now uses the definition in #1163:

It is the team repo commit the scorer file was cached from, present only when that commit holds the file byte for byte and is reachable from a remote-tracking ref.
It is not HEAD at scoring time. HEAD can hold a newer scorer that failed validation, or an unpushed commit no teammate can resolve.
When the rule gives no commit, leave teamRevision out rather than guessing.

Acceptance criteria to add:

In a fresh clone of the team repo, git cat-file -e <teamRevision> succeeds for a published label, and git show <teamRevision>:scorers/<name>.md is the rubric that scored it.
A scorer edited and pulled without a commit, or committed but not pushed, produces labels without teamRevision.

I also added #1080 as a native blocker. The body and the comment above already list it, but the link was missing.

Two things from #1148 that scorers run into:

Candidates are also in passes and run.archive. Earlier passes of a step move into its passes, and the passes of children and items a loop ran again move to run.archive. Picking succeeded agent steps from run.steps alone sees only the last pass of every loop.
An end skips pending steps in its workflow. When a workflow reaches an end or a break, stopWhere skips every pending step under it and cancels open leases. A scorer step inserted after the judged step, in the same workflow, is skipped whenever the judged step itself ends that workflow, such as a step with end: done. The scorer step needs a place an end does not reach. (Resolved in a later comment: scorer work is kept apart from run.steps.)

Superseded for judge, the key and the cost: judge is { kind: "model", tool, model?, usage? } | { kind: "human" }, the key is (scorer, rubric.sourceHash, subject, judge.kind), and usage carries cost: { amount, unit } instead of costUsd. The body states the label.

Three optional changes to the label, and two notes, on the judge's model and on where the code lives. They are not conditions for shipping this issue: its scope, criteria and blockers stay as they are. If it lands first, #1137's issues adapt to what it shipped and keep reading the labels it published.

A workflow label could leave out agent, tool, model and modelAlias. In records, labels and stats, agent will mean what a step ran as: the definition and how the tool applied it, the tool and its version, model and effort as requested and as reported, and the team revision delivered to the checkout ([feat] Record the agent each workflow step ran as #1176). The YAML key agent: still selects a definition. A workflow label sits in the run record next to the pass it judges, so readers can take all of that from the pass once [feat] Record the agent each workflow step ran as #1176 records it. Until then, keeping the flat fields is fine.
The subject could name the pass, and the rubric could have an identity without a commit. Since feat(workflow): bounded loops, parallel for_each and list expressions #1148 a step can have several passes, so subject would be { runId, step, pass }. Under [feat] Define the team revision once for run records, labels and snapshots #1163's rule teamRevision is often absent, for a scorer edited and not pushed, and then nothing says which rubric gave the label. The scorer file's hash next to it would, as [feat] Define the team revision once for run records, labels and snapshots #1163 does for runs with sourceHash: rubric: { sourceHash, teamRevision? }. The logical key becomes (scorer, rubric.sourceHash, subject): the same rubric replaces a label, an edited one adds a label, and two rubrics can be compared on the same steps.
The label could name its judge. A label from another judge model is another measurement. judge: { tool, model?, usage? }, with usage (inputTokens, outputTokens, costUsd) only as far as the tool reports it.

type ScoreLabel = {
  scorer: string;
  rubric: { sourceHash: string; teamRevision?: string };
  label: string;
  score: number;
  passed: boolean;
  source: 'workflow' | 'session';
  subject:
    | { runId: string; step: string; pass: number }
    | { platform: string; sessionId: string };
  judge: { tool: string; model?: string; usage?: { inputTokens?: number; outputTokens?: number; costUsd?: number } };
  scoredAt: string;
};

#1137 adds agent to session labels, which have no run record to read it from.

The judge's model. #1177 adds a workflow defaults.model for prompt steps. A scorer step does not take it: the judge runs on the scorer's model, or the tool's default, whatever workflow it is in.

Where the code lives. #1137 scores sessions with the same scorer files. If the definition parser, sampling, the judge prompt with the parsing of its answer, the alias lookup for model, and the label type sit outside src/workflow/ (say src/scorers/), session scoring reuses them as they are. Otherwise #1137 moves them later.

Notes for scorers from #1175 and #1180. Step labels (this issue) and session labels (#1137, #1178, #1179) stay as decided, and whole-run scoring is now #1247; these points only change how a step is judged and recorded:

The scorer goes to the driver that holds the judged pass. Discussion: Workflow Runtime Integration #818's accepted v1 leases the scorer "to the same driver so the content stays where it is", and the decision here resolves the session "like session: { from: <step> }". The body calls it "an internal read-only agent step", and in feat(workflow): add workspace branches #1175 a read-only step runs in the member's checkout (workspace.ts:242), which cannot see a [feat] Run benchmark suites as workflows #1180 trial's diff in the trial's own clone. I'd lease a scorer step only to the driver that owns the judged pass's copy, by the owner check a continue step gets (workspace.ts:260-263), with the judged pass's changes on the lease. That driver prepares the evidence where the commits and the session are, and the answer-only judge runs outside the copy, in an empty directory, as [feat] Score a finished session on demand #1178 describes. The scorer is not itself a continue step, since the local driver commits and pushes whatever a continue step leaves (worktree.ts:287). [feat] Run benchmark suites as workflows #1180's "its scorer steps continue its copy" would change to match.
Only a driver that can run a judge takes it. The judge reads text the judged agent wrote, and [Proposal] Score agent sessions and run benchmarks as workflows #1137 proposes that "the judge can only answer". Today any driver, the local one included, takes any prompt lease: driver.can reads only hitl and workspace (engine.ts:1112-1125). I'd lease scorer work only when that driver also declares can.judge, with the tools it can run as a judge. Unknown can keys are listed in ignored, so older CLIs keep working. The profile list that makes a judge answer-only is in [feat] Score a finished session on demand #1178, which is blocked by this issue; this issue would own the list, and [feat] Score a finished session on demand #1178 would reuse it and move its criterion 6 here.
A scorer no driver can take must not hold the run. A pending step keeps its workflow from settling (engine.ts:717), and an idle local driver returns busy (local-driver.ts:850). With the can.judge gate that is the normal case for a member whose tool has no profile. A scorer step that no driver can take settles without a label once nothing else in the run is pending, and the local driver settles it at once when it has no judge profile. A scorer that runs gets a timeout of its own (see [feat] Enforce workflow requirements and execution limits #1082).
Where the judge's answer goes. The decision says reasoning stays local. The proposal it adopted put it in the run state, and [feat] Publish and resume complete workflow state across machines #1084 publishes the run state as state.yaml. This issue defines no place for a workflow judge's answer. I'd write it to <dataHome>/scores/reasoning/<labelId>.md, the store [feat] Score a finished session on demand #1178 defines for sessions, and keep only the label and labelId in the run file. That needs the pass and rubric.sourceHash in the label key (point 2 of the 2026-10-16 11:08 UTC comment), and not only for the file: under the locked subject: { runId, step }, sampling and "a rerun replaces the label" treat every pass of a looped step as one subject, so labels collide in the record too. Criterion: after a scored run, a phrase that occurs only in the judge's answer appears in no file under the run directory and in no published record.
Scorer steps are not run work, but what scoring costs stays visible. The decision keeps a failed scorer out of the run's outcome; the record's other fields need the same rule. steps[], stepCounts and iterations count every step (records.ts:202-230), failureOf takes the first failed top-level step (records.ts:155-166), and the Workflow Runs page sums every step's tokens (dashboard/workflow-runs.ts:124-138). Scorer work kept apart from run.steps (later comment on this issue) stays out of all of these and of [feat] Enforce workflow requirements and execution limits #1082's budget, and keep their usage in a separate scoring total that counts every attempt, including one that fails and gives no label. A label may also carry its judge's usage (judge.usage, point 3 of the 2026-10-16 11:08 UTC comment), with cost as cost: { amount, unit } in the tool's own unit, replacing costUsd; [feat] Score a finished session on demand #1178's session label, which is this label, takes the same shape, and [feat] Run benchmark suites as workflows #1180 would use it too. Criterion: a run whose sampled scorer fails or spends tokens has the same stepCounts, iterations, failure and work token total as the same run with samplingRate: 0, and its tokens appear in the scoring total.
The body still says "Record only scorer name and label". Replace it with the locked label, and say whose tool/model it carries: the judged pass's, read from that pass's report. The judge's own tool and model go in judge if point 3 is adopted, and nowhere in the label otherwise. Without that, a reader cannot tell whether model is the judge's, and stats that sum labels per model ([feat] Publish privacy-preserving workflow run records and stats #1083) mix the two. Criterion: with a judge on another model, judge.model names the judge's model, and a reader takes the judged pass's model from that pass's agent, never from the label.
A reviewer's evidence is the range it reviewed. Discussion: Workflow Runtime Integration #818's review-quality scorer judges the reviewer, a read-only step. Its direct needs or the member's HEAD do not say what it reviewed: in Discussion: Workflow Runtime Integration #818's review-and-fix, the first review has no local needs, and the change lives in the enclosing copy. With F4 in [feat] Close the #1080 gaps that #1175 leaves open #1246, a read-only step inside a copy runs in that copy and its pass records the commit it read. The scorer's evidence is that range, the pass's outputs and its redacted session; with no recorded range, the evidence says so rather than guessing. Criterion: a scorer on the review step (agent reviewer) of Discussion: Workflow Runtime Integration #818's review-and-fix receives the implementer's diff and the reviewer's verdict.
The Workflow Runs page was promised labels. [feat] Add a Workflow Runs page to the dashboard #1088 scoped "scorer labels when present", and PR feat(dashboard): add a Workflow Runs page with HITL answers (#1088) #1225 deferred them until records have them. Add { scorer, label, passed } per pass to that page's allowlist (dashboard/workflow-runs.ts:30-69), never evidence or reasoning.

The recorded review range is F4 in #1246, which this issue now waits for.

feat(review): fail review --branch on an invalid workflow

Subagent transcripts in a step's evidence. A step's session is resolved like session: { from: <step> }, and today that leaves out the work its subagents did:

Claude Code writes each subagent to <sessionId>/subagents/agent-<agentId>.jsonl, next to the session file, with isSidechain: true on every record. The Claude adapter reads only the session file and also drops sidechain records (claude-code.ts:560).
CodeBuddy keeps the same subagents/ directory, and its adapter does not read it either ("not migrated in the first version", codebuddy.ts:10).

A step that hands its edits to a subagent is then judged on a session that did nothing. I'd have AgentSession.find read the subagent files for these tools, and have AgentSession.transcript give their entries marked with the subagent's id, in the order they ran, so the judge sees who did what. Scoring a subagent as its own subject stays out, as in #1137. Session scoring (#1178) reuses this unchanged.

Criterion: a fixture Claude session whose only edit is made by a subagent gives the step's scorer that subagent's tool call.

Proposal: a workflow lists a step's scorers, and a scorer file holds only its rubric.

agents in the scorer file selects by definition, and an agent definition is only the role file a step hands the tool, not what TeamAI delivered to it. That misses work: a prompt step without agent: cannot be scored, sessions needed sessions: true (#1179), delivering steps needed selects (#1247), and one role used in two workflows for different goals gets the same scorer. The label already records the whole context the pass ran with (#1176, team.selection and used.skills in #1178), so grouping by definition keeps working from agent.definition. Instead:

A prompt or workflow step takes scorers: [<name>, ...]. On a workflow step the scorers judge everything under it. On a for_each step they judge each item. There is no workflow-level key: the delivering step already covers the run ([feat] Score what a workflow run delivered #1247). Pull reports a name with no valid scorer, and scorers: on a step that runs no agent. teamai review --branch reports the same through [feat] Validate workflows, prompts and scorers in review --branch #1248's validator.
The scorer file keeps description, labels, passingScore, samplingRate, model and the prompt body, plus sessions ([feat] Score finished sessions in the background #1179) and check ([feat] Score what a workflow run delivered #1247). It drops agents. A file that still has agents or includeDescendants, as a ported Warp scorer would, or selects, from an earlier draft of [feat] Score what a workflow run delivered #1247, fails validation with scorers/<name>.md uses `<key>`. A workflow lists the scorers for a step in its `scorers:` key.
The subject is one succeeded pass: { runId, step, pass }. Under { runId, step }, every pass of a looped step is one subject, and an outer loop that reruns a sub-workflow reuses the same path and continues its pass numbers (archiveUnder, firstPass in loop.ts), so labels of different rounds collide. Each pass also reviews or changes a different range. To judge a whole loop or fan-out, list the scorer on the workflow step that contains it. A prompt step's label reads agent from its pass. A workflow step's label has no agent, since its passes usually mix definitions.
Scorer work is kept apart from run.steps, one entry per sampled (scorer, subject), leased through next. In run.steps it would be skipped by a top-level end (stopWhere matches every id under the top scope's prefix '', scope.ts), moved by archiveUnder, and counted by stopped() and the record totals. Entries are independent of each other and of the run's outcome. The end record is written once every entry has settled, labelled or not, as [feat] Run benchmark suites as workflows #1180 asks for trial scorers.
Several scorers on one step never wait for each other. Whether their judges overlap is the driver's choice: the local driver runs one prompt lease at a time (local-driver.ts), so it runs them one after another.
Edge attribution is no longer needed: a scorer judges the step it is listed on, and a reviewer's evidence is the range its pass recorded (F4 in [feat] Close the #1080 gaps that #1175 leaves open #1246).

Body changes this implies (applied to the body):

"Add and validate scorers/*.md definitions with agents, labels, scores, passing score, sampling rate, model, and prompt." becomes "Add and validate scorers/*.md definitions with labels, scores, passing score, sampling rate, model, and prompt, and the scorers: step key that names them."
"Select sampled successful agent steps deterministically." becomes "Select sampled succeeded passes of the steps that list a scorer, deterministically."
Remove "Attribute a scorer to the graph edge it judges when unambiguous."
"Surface ambiguous attribution and malformed definitions during pull." becomes "Surface malformed definitions and unknown scorers: names during pull."
In "Out of scope", remove "explicit evaluates".
"Sampling is deterministic for a run/step/scorer tuple." becomes "Sampling is deterministic for a run/step/pass/scorer tuple."

Body updated after a design pass. What changed:

Before	Now	Why
Lease scorer work through `next/report`	`teamai score --next` hands out judge work; `workflow next` never does	a judge lease has no step, and the engine cancels attempts it cannot find (`engine.ts:890-897`)
Label in the run record; end record waits for scorers	one label store, `scores/<member>/<YYYY-MM>.jsonl`, shared with #1178; the run settles first	a late label moved `endedAt` and raced the outbox
Scorer list stored next to `run.steps`	open work derived from the run, labels and attempts	the engine gets no scoring code path, so counts cannot drift
`can.judge`	`can.judge: { isolation, tools }`	choosing the judge model needs the tool

Judge profiles, measured with a hostile fixture transcript:

Tool	Access	Level
Claude (`--restricted --tools=Read,Grep,Glob`, cwd = evidence)	`read`	`verified`
Codex (`exec`, permission profile: read evidence only, network off, `[agents] enabled = false`)	`read`	`verified`
OpenCode 2 (`--standalone`, isolated config and db, project config off, evidence outside any checkout)	`read`	`verified`
CodeBuddy (Claude-like flags, no `--restricted`)	`read`	`best-effort` until a run proves it

Any driver judges the same way:

$ teamai score --next --json '{"driver":{"kind":"t3","machine":"m1","can":{"judge":{"isolation":"best-effort","tools":["claude"]}}}}'
$ teamai score --lease s5d1e07c2a9b4/a91c0d7e --submit --stdin < answer.json

test(push): use .md scorer names in the placement samples

feat(namespaces): resolve inactive scorers namespaces like prompts

feat(push): support all namespaced resource types (#1234)

Implementation note from #1297: the team's scoring settings live next to the scorers, by namespace, like model aliases, env, hooks and MCP, instead of in teamai.yaml:

scorers/scoring.yaml        # every member
scorers/<ns>/scoring.yaml   # members with <ns> active; replaces the root key of the same name

# scorers/scoring.yaml
isolation: verified
batch: true

Two active namespaces setting isolation: the strictest wins. Setting batch differently is an error for those members until one file changes.
A scorer's own isolation and the machine's scoring.isolation in config.yaml still apply; the strictest wins.
batch: a step's scorers.batch, else the workflow's defaults.scorers.batch, else scoring.yaml.
A scorers/<dir>/ that no role or project declares is shared, as for prompts/.

Navigation Menu

Repository navigation

[feat] Add workflow scorers and recorded quality labels #1085

Goal

How it works

Scope

Out of scope

Dependencies

Acceptance criteria

Verification

2 remaining items

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

[feat] Add workflow scorers and recorded quality labels #1085

Description

Goal

How it works

Scope

Out of scope

Dependencies

Acceptance criteria

Verification

Activity

SaulMoro commented on Oct 13, 2026

hsuchifeng commented on Oct 14, 2026

SaulMoro commented on Oct 14, 2026

SaulMoro commented on Oct 14, 2026

hsuchifeng commented on Oct 15, 2026

SaulMoro commented on Oct 15, 2026

SaulMoro commented on Oct 16, 2026

SaulMoro commented on Oct 16, 2026

2 remaining items

SaulMoro commented on Oct 20, 2026

SaulMoro commented on Oct 22, 2026

SaulMoro commented on Oct 22, 2026

SaulMoro commented on Oct 22, 2026

SaulMoro commented on Oct 23, 2026

Metadata

Metadata

Assignees

Labels

Type

Projects

Milestone

Relationships

Development

Issue actions