[feat] Compare scorers with hand labels, including on pushed sessions #1269

Parent: #1137

Goal

A scorer is an LLM judge, and nothing measures whether its labels are right. #1249 will open PRs from repeated scorer failures, so a judge that is off makes those PRs chase noise. Let a member label a session by hand with a team scorer, on their own session or on one a teammate pushed with teamai session push, and show how often the model judge agrees with them.

Today no scorer reaches a pushed session: #1178 scores only sessions the tool keeps on this machine. This issue adds that source for the model judge and for hand labels alike.

Example

inside the session, after the tool missed a bug:
$ teamai score --scorer review-quality --label missed-bug --push
session 7f3c (claude-code)  "fix the retry loop in sync.ts"
review-quality  missed-bug  score 0  failed  (judge: human)

later, a teammate's pushed session:
$ teamai session list
  SESSION                              AUTHOR  PLATFORM     MSGS  UPDATED
  claude-code_fix-retry-loop_20261020  alice   claude-code  84    2d ago
$ teamai score --team-session claude-code_fix-retry-loop_20261020 --scorer review-quality --label missed-bug --push
session 7f3c (claude-code, alice)  "fix the retry loop in sync.ts"
review-quality  missed-bug  score 0  failed  (judge: human)

dashboard  review-quality  rubric 3a9e1c  source session
  judge                      labels  paired  agree  judge passed, human failed  judge failed, human passed
  claude / <recorded-model>  42      12      10     2                           0
  human                      12

Scope

Hand labels

• --label <value> on teamai score ([feat] Score a finished session on demand #1178). <value> must be one of the scorer's labels, or the command fails and lists them. No judge runs. The label is [feat] Score a finished session on demand #1178's, with judge: { kind: "human" }, score and passed taken from the scorer as for a model label, no reasoning file, and the same store, outbox and --push.
• A hand label and a model label coexist, through [feat] Score a finished session on demand #1178's judge.kind and its logical key (scorer, rubric.sourceHash, subject, judge.kind). Rescoring with the model never replaces the member's label, and a new hand label never replaces the model's.

Which session

• The current session or a native one, as [feat] Score a finished session on demand #1178 resolves them for teamai score. From inside the session, the tool's session variable names it, which is the main path for the author. Otherwise --session <id> names it. The command never guesses.
• A pushed session. --team-session <sessionName>, with --author <name> when two members pushed a session with that name, as session resume takes them. It reads the session with loadSession from the team repo the session commands use, so the member runs session pull first for the latest. It works with --label and with the model judge, and cannot be combined with --session.
• One subject per session. A pushed session's subject is { platform, sessionId } from its meta's origin, so a label on alice's pushed copy and one from her machine on her native session are about the same session, and they pair.
• Redacted for the judge. A pushed session may have been archived without --scrub, so it goes through the same redaction as AgentSession.find before it reaches the model judge.
• No agent on a pushed session. A capture exists only on the machine that ran the session ([feat] Score a finished session on demand #1178), so its labels have no agent, and the dashboard groups them by platform only.
• For a pushed session, the command also prints its author before the label, next to the first prompt and directory [feat] Score a finished session on demand #1178 prints.

Agreement in the scorer view (#1182)

For each scorer, rubric and source, each model judge row shows how many of its subjects also have a hand label, how many agree on passed, and the two kinds of disagreement. Several members' hand labels on one subject share #1178's key, so only the newest one counts. A hand label records the member's judgment of the session, which does not change when the judge's prompt does. The newest hand label on a subject pairs with the model labels of every rubric of the same scorer that still lists its value, and each model rubric row shows its own counts. A hand label whose value the rubric no longer has is not paired with it. When the scorer's labels or passingScore change, the guide says to label again.

Out of scope

Hand labels on workflow steps and benchmark trials, a statistic beyond counts, a minimum agreement gate (#1249 shows agreement and never gates on it), background scoring of pushed sessions (#1179), scoring a whole session list in one command, and showing session ids or writing hand labels in the dashboard (its session table keeps the id only in data-session today).

Dependencies

• Blocked by: [feat] Score a finished session on demand #1178 (session labels and their store), [feat] Show scorer results in the dashboard #1182 (the scorer view), [feat] Publish and score sessions by run mode #1385 (where pushed sessions live)
• Unblocks: [feat] Open improvement PRs proven by a before/after benchmark #1249, which shows agreement in its PR body

Acceptance criteria

1. Inside a Claude Code session with CLAUDE_CODE_SESSION_ID set, teamai score --scorer <name> --label <value> --push writes one label for that session with judge.kind: "human", launches no judge, and publishes it to scores/<member>/<YYYY-MM>.jsonl.
2. An unknown --label value fails, lists the scorer's labels, and writes nothing. --label without --scorer fails before anything runs.
3. Scoring the same session again with the model leaves both the hand label and the model label on origin.
4. On bob's machine, the example's --team-session command labels alice's pushed session with her origin.platform and origin.sessionId. The same command without --label gets a model label for that subject.
5. A pushed session archived without --scrub, holding a fixture token, reaches the judge with the token redacted.
6. An unknown --team-session name fails and points to teamai session list. An ambiguous one fails and lists its authors. --team-session with --session fails before anything runs.
7. With 12 sessions labelled both ways, 10 of them agreeing, the view shows paired 12 and agree 10 and splits the 2 disagreements by direction. A hand label on alice's pushed copy pairs with the model label from her own machine.
8. After the scorer's prompt body changes and the model scores the same 12 sessions again, the new rubric's row shows paired 12 against the existing hand labels. After a label value is removed, hand labels with that value pair with no label on the new rubric.

Verification

Unit tests for the key, the label shape, subject resolution from a pushed meta, the flag checks and pairing; e2e with a fixture session and a fixture team repo for criteria 1 to 6; one real-CLI run recorded in the PR. Update the scorer guide in both languages and regenerate skill-data/core/references/commands.md.

Update: where pushed sessions live

--team-session <sessionName> resolves the archive as the session commands do (resolveRepoRoot and sessionArchiveRoot, src/session-flow/session-cmd-shared.ts:213-250). It reads teamai-reports first (#1385), then, on a git-kind team, the old archive on the default branch, read-only. A self-mode team's archive is already on teamai-reports. A name resolves in teamai-reports first; the old archive is read only when the name is absent there, and the --author ambiguity check runs within the location where the name was found. session pull refreshes both locations (#1385).

$ teamai session pull
$ teamai score --team-session claude-code_fix-retry-loop_20261020 --author alice --scorer review-quality --label missed-bug --push
session 7f3c (claude-code, alice)  "fix the retry loop in sync.ts"
review-quality  missed-bug  score 0  failed  (judge: human)

Added criteria:

9. A session pushed before [feat] Publish and score sessions by run mode #1385, which sits on the default branch of a git-kind team, is still found by --team-session.
10. A name that exists on teamai-reports and in the old archive resolves to the teamai-reports copy, and two authors of that name on teamai-reports fail as ambiguous, listing those two authors only.