[feat] Judge a session with the driver's own tool #1386

Example

The caller's agent judges the session itself, for example in a subagent or a separate agent it launches: it takes the lease, gives that judge prompt.md and the evidence directory, and submits the judge's answer. teamai launches nothing on this path.

agent session (Cursor, Claude Code, an orchestrator such as T3…)
  teamai score --scorer … --session … --json '{…"can":{"judge":{…}}}'   → lease: prompt.md, evidence, answer schema, submit command
  subagent or separate agent: reads only prompt.md and the evidence      → {"label":"good","reasoning":"…"}
  teamai score --lease <ref> --submit --stdin                            → label in scores/<member>/<YYYY-MM>.jsonl

A member works in Cursor. teamai reads Cursor sessions, but cannot launch Cursor as a judge. Their Cursor session takes the lease and judges it in a subagent:

$ teamai score --scorer review-quality --session 7c1e… --source cursor \
    --json '{"driver":{"kind":"cursor","machine":"m1","workspace":"<workspace>","can":{"judge":{"isolation":"best-effort","access":"read","tools":["cursor"]}}}}'
{"data":{"lease":{"ref":"s5d1e07c2a9b40/a91c0d7e0","source":"session","scorer":"review-quality",
  "subject":{"platform":"cursor","sessionId":"7c1e…"},"cwd":"<project-directory>",
  "evidence":"…/scores/evidence/a91c0d7e0/","prompt":"…/prompt.md","answer":{"schema":{…}},
  "submit":"teamai score --lease s5d1e07c2a9b40/a91c0d7e0 --submit --stdin"}}}

$ teamai score --lease s5d1e07c2a9b40/a91c0d7e0 --submit --stdin <<'JSON'
{"driver":{"kind":"cursor","machine":"m1","workspace":"<workspace>"},"outcome":"success",
 "answer":{"label":"good","reasoning":"…"},"judge":{"tool":"cursor"}}
JSON
{"data":{"label":{"scorer":"review-quality","label":"good","score":1,"passed":true,…},"published":true}}

The stored label in scores/<member>/<YYYY-MM>.jsonl records the judge:

"judge": {"kind": "model", "tool": "cursor", "isolation": "best-effort", "access": "read"}

Without --json, nothing changes: the local judge runs as today, with a judge tool installed on this machine.

$ teamai score --scorer review-quality --session 7c1e…
ℹ Scoring claude-code session 7c1e… in <project-directory>

A team that requires verified isolation:

$ teamai score --scorer secure-review --session 7c1e… --source cursor --json '{"driver":{…,"can":{"judge":{"isolation":"best-effort",…}}}}'
{"error":{"code":"isolation_unmet","message":"scorers/secure-review.md asks for verified isolation, and this driver offers best-effort. No judge work was taken."}}

A scorer that pins a model, with a judge tool that is not a recognized tool id:

$ teamai score --scorer review-quality --session 7c1e… --source cursor --json '{"driver":{…,"can":{"judge":{…,"tools":["my-judge"]}}}}'
{"error":{"code":"judge_model_unavailable","message":"…"}}

Behavior

Which path a driver should use for a session:

This table is routing advice. The CLI returns a lease for any session when --json declares can.judge, whatever its tool.

The judge tool is not the session's platform. --source names where the session is read, a platform teamai has a session adapter for (teamai session list shows them). can.judge.tools names the tools the driver runs its judge with.

Skill driver without a confined judge

Today (#1304) a skill-driven run in Cursor reaches done with its pass judge work left open, because teamai has no confined Cursor judge. After this issue the driver judges that work itself:

 describe → judging
   here        → teamai runs the confined judge (#1304), unchanged
-  elsewhere   → the judge work stays open for another driver's score --next
+  elsewhere   → the driver takes its own run's judge work:
+                teamai score --next --json '{"run":"r3f9…","driver":{…,"can":{"judge":{"isolation":"best-effort","access":"read","tools":["cursor"]}}},"running":[]}'
+                a subagent judges each lease from prompt.md and the evidence only
+                teamai score --lease <ref> --submit --stdin
+                work a scorer requires verified for stays open for another driver, as today

turn   score --next answer                     driver does
1      lease s5d1…/a91c… (review-quality)      subagent judges it, submit → label best-effort
2      lease s5d1…/b2e0… (style)               subagent judges it, submit → label best-effort
3      idle                                    stops; secure-review (verified) is left open

Acceptance criteria

• The explicit request leases through takeJudgeLease: the same attempt store, lease ref, evidence directory, redaction, answer schema, submit path and labels as score --next. Its entries come from the named session and scorers, resolved as score --session resolves them today (session, scope, pins). There is no second judging path.

• This deliberately changes the entries invariant "Never a session" (JudgeRequest, src/workflow/scoring.ts): an explicit request may now name a session entry. The background driver's selection is unchanged.

• The label records judge.tool (the lease's tool, which the submit may repeat), judge.access from the request's can.judge.access (default full), and judge.model.reported from the submit's judge.model.

• isolation_unmet at lease time is new behavior, covered by a test. Before this change it existed only on submit.

• A model-pinned scorer with can.judge.tools: ["cursor"] and a literal model name gets a lease. Covered by a test with an empty alias map.

• --session with --json is accepted for this request; the invalid_arguments message changes accordingly. The error rows in the input table are covered by tests.

• Without --json, score --session behaves exactly as today, its error codes included.

• Skill driver, judging elsewhere: skill-data/workflow/references/judging.md tells the driver, after done, to take its own run's judge work with score --next and run, declaring isolation: best-effort and its session's tool, to judge each lease in a fresh subagent that reads only prompt.md and the evidence directory, to submit each answer, and to stop at idle. With judging here, the confined judge stays the only path: a driver never downgrades verified judging to best-effort.

• A lease taken across projects is found again. From project B, score --next with run finds a run of project A through findRunConfig and leases there (runConfig, src/score-commands.ts), but score --lease <ref> and --submit look only in this directory's scope and the user scope (leaseConfig, same file), so neither finds that lease. leaseConfig resolves a lease in the same scopes runConfig searches. Covered by a test that takes the lease from project B and judges and submits it from B. A driver that follows the lease's cwd already succeeds today; this makes a submit from B succeed as well, and lease_unknown's message stays true. Carried over from feat(workflow): add session discovery and managed public leases #1304's last review, merged unfixed.

• describe offers "Same, but without judging" for judging elsewhere too, and a run started with --no-judge takes no judge work. Covered by a test of the describe answer for a session tool with no confined judge.

Definition of done

• docs/guide/scorers.md and docs/guide/zh-CN/scorers.md: "Score a session" gains the driver example; the "Any driver" section and the isolation table link to it.
• score help text: --session with --json and can.judge, and the filters an explicit request skips. skill-data/core/references/commands.md regenerated with npx vitest run commands-reference -u. The skill body does not change.
• Real-CLI record: one lease taken and submitted for a captured session with a --json driver.

Out of scope

• When scoring is offered or automatic ([feat] Publish and score sessions by run mode #1385, [feat] Default workflows a team can override, starting with session-close #1388). The friction reminder is [feat] Default workflows a team can override, starting with session-close #1388's and is not changed here.
• Background judging, unchanged.
• Reading sessions of a tool teamai has no session adapter for (Pi, for example). Such a session cannot be scored until an adapter exists.
• New judge tools teamai launches itself. [feat] One launcher for every AI call, from the tool descriptor #1389 adds none either.

Notes

• Session demand: sessionDemandOf (src/score-commands.ts:75-86) rejects a payload today. score routes at :118-161. Session resolution and evidence: src/session-flow/scoring.ts:57-117 (sessionOf, scopeOf, scoreSession). Today's local judge choice: installedJudgeTools([]) and chooseJudgeModel (src/session-flow/scoring.ts:124-132).
• One lease path with a second entry source: scoreOnDemand already passes entries into takeJudgeLease for passes (src/workflow/scoring.ts). The background filters live in entries (src/session-flow/session-work.ts:256) and in takeOneLease, which also checks the daily limit (src/workflow/scoring.ts). A held entry answers { kind: 'busy' } there.
• Access is taken from the request at lease time, in takeOneLease (src/workflow/scoring.ts); the submit payload, SubmitPayloadSchema, has no access. The submitted tool must match the lease's (labelMembers).
• Isolation today: takeOneLease filters leasable entries by meets (src/workflow/scoring.ts); the submit-time isolation_unmet is in labelMembers.
• Judge tools teamai launches: src/scorers/judge.ts:36. can.judge.tools takes any string (NextPayloadSchema, src/workflow/scoring.ts). A model-pinned scorer resolves only for recognized tool ids (src/scorers/judge-model.ts:38-47, isAgentToolId in src/tools.ts:856-860), through resolveModelValue, which takes a name the alias map does not hold as a literal model (src/models/aliases.ts:566). This issue keeps that rule.
• Session platforms teamai reads: src/session-flow/adapters/platform-facts.ts. An unsupported --source passes sessionOf (src/session-flow/scoring.ts:63-64); AgentSession.find then fails no_adapter (src/session-flow/agent-session.ts:145-152), and scoreSession returns it as session_not_found (src/session-flow/scoring.ts:94-95). This issue keeps that public code.
• The skill driver's judging today (feat(workflow): add session discovery and managed public leases #1304): describe resolves judging (resolveJudging, src/workflow/protocol.ts). here means teamai runs the confined judge of the session's tool; elsewhere means it has none (Cursor: "has no confined judge teamai runs"), and the run's pass judge work stays open in that data home for another driver's score --next, which teamai score --drain reports as left open ([feat] Publish and score sessions by run mode #1385). score --next already takes run (NextPayloadSchema, src/workflow/scoring.ts), and takeOneLease leases only that run's entries when it is set, so the driver's own-work loop needs no new command or flag. The "Same, but without judging" option is built only for here today (src/workflow/drivers/skill/preflight.ts, optionsOf). This issue's explicit --session lease is a different entry source and is not used for passes.
• Lease ref format: s<13 hex>/a<8 hex> (src/scorers/entry.ts:16-20).
• Docs to extend: docs/guide/scorers.md:101-128 ("Any driver").