[Proposal] Judge recall adoption with a session scorer #1391

Example

Today, when a member sets TEAMAI_UPVOTE_JUDGE=1, each Stop asks the local CLI which recalled documents the final reply used without opening them, and upvotes those. Opening a recalled document is credited without any judge, and that path is unchanged by this proposal.

session 7c1e… recalled: [pod-startup-timeouts, helm-values, retry-policy]
opened by a tool call  → credited without AI: [helm-values]
votes-judge (opt-in)   → adopted: [pod-startup-timeouts]   → upvote

As a scorer, the same judgment would go through sampling, leases, any driver's tool and isolation, and land as labels. A valid scorer file, as the scorer contract stands today:

---
# scorers/recall-adoption.md
labels:
  - { value: adopted, score: 1, description: The final reply substantively used a recalled document. }
  - { value: ignored, score: 0, description: No recalled document shaped the final reply. }
passingScore: 1
sessions: true
samplingRate: 25
---
Read the session transcript in the evidence. Did the final reply substantively use any document that recall returned in this session?

{"scorer":"recall-adoption","subject":{"platform":"claude-code","sessionId":"7c1e…"},"label":"adopted","score":1,"passed":true}

This gives one label per session, not per document. Naming the document in the prompt would need per-subject prompt input, which scorers do not have.

Behavior

The rows are not independent. "Upvote from published labels" (Upvotes row) needs the per-document subject or the list answer (first row): one label per session names no document to upvote.

Decision: the maintainers choose in this issue. Choose a scorer option only if per-document upvotes stay possible (first row) and recall confidence keeps its inputs (Sampling row); otherwise keep votes-judge.

Acceptance criteria (if accepted)

• For the scorer options: votes-judge, its direct CLI call and its hook handler are removed, and the TEAMAI_UPVOTE_JUDGE text leaves the docs. Nothing calls an AI tool for adoption outside scorers.
• For the "keep" option: votes-judge stays opt-in and calls the launcher from [feat] One launcher for every AI call, from the tool descriptor #1389.
• Recall confidence keeps its current inputs or a documented replacement.
• Grounding stays as strict as today: knowledge roots only, no symlinks, fail closed, one upvote per document per session.

Definition of done

• docs/guide/scorers.md, docs/guide/knowledge.md and their zh-CN pages, for the option chosen.
• skill-data/core/references/troubleshooting.md: the TEAMAI_UPVOTE_JUDGE text updated or removed.
• Real-CLI record: one session judged for adoption, with the upvote it produced.

Out of scope

• Crediting documents opened by a tool call, which needs no judge and stays as it is.
• Making adoption judging on by default in attended sessions. In unattended runs, [feat] Publish and score sessions by run mode #1385's default (unattended: auto) applies to a scorer with sessions: true like any session scorer, so a team that adds this scorer gets adoption judging there unless it sets unattended: off.

Notes

• votes-judge: src/votes-judge.ts ([feat] upvote: collect adoption without relying on AI self-declaration (follow-up to #369) #723, Proposal: attribute recall and adoption to the agent session in every agent #884), grounding at :103-133 and :185-200. Hook handler: src/hook-handlers.ts:729-869, opt-in check at :734. Tool-use credit: creditAdoptedDocs (src/hook-handlers.ts:593, :763).
• Scorer contract: src/scorers/definition.ts:15-45, docs/guide/scorers.md:9-31. The example above parses with parseScorer. Session subject: SessionSubjectSchema (src/scorers/label.ts:13).
• Confidence: src/maintenance/confidence.ts:27-49.
• Docs that name the opt-in: skill-data/core/references/troubleshooting.md:347-352.
• Session scorers and score --next leases for any driver exist today; this proposal depends on no open issue for them.