[feat] Score what a workflow run delivered #1247

Parent: #1137

Goal

Answer "did this PR deliver what was asked, and only that?" for a workflow run, next to the step labels of #1085 and the session labels of #1178 and #1179, which stay as they are.

Example

# scorers/delivered-what-was-asked.md (front matter)
check: { run: npm test, protect: ['test/**', package.json] }   # optional; #1180's check
labels:
  - { value: complete, score: 1, description: Delivers what was asked and nothing else. }
  - { value: partial, score: 0.5, description: Delivers part of what was asked. }
  - { value: off-target, score: 0, description: Delivers something other than what was asked. }
passingScore: 1

# workflows/ship-issue.yaml (excerpt)
  - id: change
    workspace: new
    deliver: pr
    scorers: [delivered-what-was-asked]   # judges what this step delivered
    workflow: { ... }

run ship-issue (issue: 42)
  read-issue   outputs { title, body: "Add pagination to /users" }   ← stored in the run at report time
  change       delivers PR #57
  scorer       CI on #57: red  →  label partial, decidedBy: ci     (judge not called)

issue #42 edited later to "Add pagination and filters"  →  the label does not change; evidence comes from the run

Scope

• A scorer listed in scorers: ([feat] Add workflow scorers and recorded quality labels #1085) on a step that delivers (deliver: pr), including a workflow step, judges what that step delivered, with the rules below.
• Deterministic checks decide first, as [feat] Run benchmark suites as workflows #1180 grades with a check before any judge: an optional check in the scorer file runs on the delivered branch with [feat] Run benchmark suites as workflows #1180's check semantics, then the delivered PR's CI status if it has finished when the scorer runs (a pending status counts as absent; this issue adds no wait), and the protected-path result ([feat] Deliver workflow branches as pull requests #1090). The judge runs only when they pass or are absent. The label adds decidedBy: check | ci | protected | judge. A label decided by check, ci or protected still names the scorer's model judge, judge: { kind: "model", tool, model? }, with no usage, so the scorer's pass rate stays one measurement whichever stage decided.
• Evidence, built by the driver that holds the delivered branch and kept local: the delivered diff, the run's inputs, the outputs every step reported (stored in the run when each step reported, so later edits to an issue do not change them), and the redacted sessions of the agent passes under the step. No new key: a workflow that wants the request in the evidence has a step return it as a typed output, as read-issue does above. The label stores the hash of the evidence.
• The label uses the [feat] Add workflow scorers and recorded quality labels #1085 record with subject: { runId, step, pass }, with pass as [feat] Add workflow scorers and recorded quality labels #1085 defines it, so two PRs from two passes stay two measurements. It keeps the judge's usage (judge.usage, with cost as cost: { amount, unit } when the tool states one).
• A scorer that no driver can take, or that times out, settles without a label and does not change the run (as proposed on [feat] Add workflow scorers and recorded quality labels #1085 and [feat] Enforce workflow requirements and execution limits #1082).
• Sampling is [feat] Add workflow scorers and recorded quality labels #1085's, over subject: { runId, step, pass }.
• On demand. teamai score --run <id> [--step <name>] [--pass <n>] --scorer <name> ([feat] Score a finished session on demand #1178's command) scores one pass, ignoring sampling, as Warp scores any single run on demand. It takes any scorer, for any succeeded pass of a prompt or workflow step. --step can be left out when the run's workflow lists that scorer on exactly one step, and --pass defaults to the last pass. The evidence is rebuilt from what the run recorded: the pass's commits, the step outputs stored in the run, and its sessions. It runs on the machine whose driver ran the pass. When the commits or a session are no longer there, for example after the branch that held the pass's commits was deleted, the command fails, names what is missing, and writes no label.

Out of scope

A request key in the workflow file, reading the work item again at scoring time, scoring a run that ran on another machine, scoring runs that deliver nothing, and dashboard views beyond #1182.

Dependencies

• Blocked by: [feat] Add workflow scorers and recorded quality labels #1085, [feat] Deliver workflow branches as pull requests #1090, [feat] Run benchmark suites as workflows #1180, [feat] Score a finished session on demand #1178 (the teamai score command)
• Unblocks: none

Acceptance criteria

• In the example, a red CI on the delivered PR gives partial with decidedBy: ci, and no judge runs.
• With CI green and no check, the judge receives the delivered diff, inputs, every step's outputs and the redacted sessions, and the label stores the evidence hash.
• Editing the issue after delivery and scoring again gives the same evidence hash.
• Two passes that deliver two PRs give two labels with different subject.pass.
• teamai score --run <id> --scorer <name> on a finished run with samplingRate: 0 gives a label for its delivering pass. On a run whose session is gone, it fails, names the session, and writes no label.
• No prompt, diff, transcript, output or reasoning appears in a published record.

Verification

Focused check-first, evidence and label tests; build/typecheck/lint; one real-CLI run of the example against a fixture repo with a fake provider. Bilingual scorer docs with the example.