[feat] Publish and score sessions by run mode #1385

Example

A team that sets nothing gets these defaults:

So a person's sessions behave as today, and an unattended job publishes and judges its own sessions before it exits. The local driver drains at the end of the run:

$ CI=true teamai workflow start review-and-fix --driver local --tools default=claude --auto-approve
…
✔ Run r3f9… succeeded
ℹ Draining before exit (teamai score --drain)
✔ Judged 2 sampled passes here (review-quality)
✔ Published 2 labels to teamai-reports
✔ Archived 2 step sessions of r3f9… to teamai-reports (scrubbed)
✔ Reported usage to stats/<member>.yaml
✔ Nothing left to do here

Every other driver ends with the same command: the skill driver in an unattended session, T3, Paseo, a CI script.

$ teamai score --drain
✔ Judged 1 sampled session here (review-quality)
✔ Published 1 label to teamai-reports
✔ Archived 1 captured session to teamai-reports (scrubbed)
✔ Reported usage to stats/<member>.yaml
ℹ Left open: 1 session whose scorer pins a model no judge tool here runs (judge_model_unavailable). A driver whose tools run it takes it with: teamai score --next --json '{"driver":{…,"can":{"judge":{…}}}}'
$ echo $?
0

$ teamai score --drain           # the judge fails
✖ review-quality on claude-code session 7c1e…: judge_failed. A failed judgment is final and is not retried. To ask for a new one: teamai score --session 7c1e… --source claude-code --scorer review-quality
✔ Reported usage to stats/<member>.yaml
$ echo $?
1
$ teamai score --drain           # run again
✔ Nothing left to do here
$ echo $?
0

$ teamai score --drain           # the team repo cannot be pushed to
✔ Judged 1 sampled session here (review-quality)
✖ Could not publish 1 label and 1 session to teamai-reports: <git error>. They stay pending; run teamai score --drain again.
$ echo $?
1

Where the drain runs decides what it sees of the calling session:

CI job:  claude -p "…"  →  tool exits  →  teamai score --drain      # the job's last step: the session is complete
skill:   … last turn of an unattended session → teamai score --drain  # the session as it stands now; later turns are
                                                                        # neither judged nor published again

This changes behavior for a team that already runs teamai in CI or another unattended job. CHANGELOG.md gets this entry under ## [Unreleased] › ### 💥 Breaking Changes, and the guide says the same, with the opt-out:

- **Unattended runs now publish and score their sessions by default.** In CI, on a cloud VM or in a hosted job
  (`CI` or `TEAMAI_MODE=unattended`), `teamai score --drain`, which the local driver runs at the end of a run and
  every other driver runs before it ends, archives to teamai-reports, scrubbed: the captured sessions of this machine
  recorded as unattended, and the step sessions of the workflow runs driven here. Every scorer with `sessions: true`
  judges a sample of those captured sessions, within daily_limit. To keep the old behavior, set `unattended: off`:
    teamai.yaml            sessions: { publish: { unattended: off } }
    scorers/scoring.yaml   sessions: { unattended: off }

And under ### Changed:

- `teamai session push`, `session migrate` and `workflow publish --sessions` archive sessions on teamai-reports on
  every team, as self mode already did. Sessions archived on the default branch before stay readable there.

The team files take a policy per mode. The boolean form of sessions keeps working:

# scorers/scoring.yaml, or scorers/<namespace>/scoring.yaml
sessions: { attended: auto, unattended: auto }   # true = auto in both modes, false = off in both
daily_limit: 10                                  # per data home: on a fresh CI runner it caps one job, not one day

# teamai.yaml
sessions:
  publish: { attended: ask, unattended: auto }   # the defaults, written out

A member's own config.yaml wins over both, one mode at a time. A mode it does not name falls through to the team file:

sessions:
  publish: { attended: ask }        # offer to publish my sessions (#1388); unattended still follows the team
scoring:
  sessions: { attended: ask }       # offer to score my sampled sessions

A bad value is reported and ignored:

$ teamai doctor
⚠ teamai.yaml sessions.publish.unattended is "yes", not off, ask or auto, so it is ignored
⚠ Sessions are published and scored in unattended mode, but teamai reads no sessions from pi: they are neither published nor judged

Behavior

teamai score --drain, a new flag on the existing score command:

Acceptance criteria

• The settings above parse in teamai.yaml, scorers/scoring.yaml, namespace scoring.yaml and member config.yaml. A bad value is reported and ignored, as src/scorers/settings.ts does today.
• With no settings, attended sessions are neither published nor judged, as today. Covered by a test with no CI and no settings.
• With no settings, an unattended job's drain archives its captured sessions and its runs' step sessions, scrubbed, to teamai-reports, and judges the captured sessions with every scorer that sets sessions: true, sampled and within daily_limit. Covered by a test with CI=true and no settings.
• unattended: off in sessions.publish and in scoring.yaml sessions restores today's behavior in an unattended job. Covered by the same test with both set.
• The boolean form of sessions in scoring.yaml and scoring.sessions in config.yaml keeps its meaning (true = auto in both modes, false = off).
• A partial policy object ({ attended: ask }) leaves the other mode to the next level.
• Publishing always scrubs. There is no setting that publishes unscrubbed.
• teamai session push --session <id> --source <platform> --scrub pushes exactly that session (today push takes the most recent --limit sessions). --source takes a session platform (claude-code, codebuddy-ide, …), as session push and pull do today. auto publishing of a captured session and session-close ([feat] Default workflows a team can override, starting with session-close #1388) use it. A run's step sessions are published with archiveSessions, the writer workflow publish --sessions already uses.
• teamai score --drain does the steps in the table above, with those exit codes, in either mode. In attended mode it applies the attended policies. Covered by tests for: nothing to do (exit 0), an entry left open as judge_model_unavailable (exit 0, printed), a failed publish (exit 1, retried by the next drain), a failed judgment (exit 1, printed, and a second drain exits 0 without judging it again), and a second drain that finds nothing new.
• The drain reports usage as teamai pull does. Covered by a test: an unattended session drained on a data home that never runs pull reaches stats/<member>.yaml under unattended.
• In attended mode the drain publishes only attended sessions idle for 6 hours. Covered by a test with publish: { attended: auto } and one active attended session, which stays unarchived.
• The local driver calls the same function as --drain at done when its mode is unattended. There is one drain implementation.
• skill-data/workflow/SKILL.md (feat(workflow): add session discovery and managed public leases #1304) gains one line in its unattended paragraph: run teamai score --drain before your session ends. done.judging keeps feat(workflow): add session discovery and managed public leases #1304's shape, and workflow next keeps never handing out judge work.
• sessionArchiveRoot returns the teamai-reports checkout for every git-kind and self-mode team. openSessionArchive (restore) and alreadyArchived read there first, then the legacy location. A step that continues a session another machine archived before this change still restores it.
• In unattended mode, a session never gets a label before a drain reaches it. Covered by a test that runs Stop three times on a TEAMAI_MODE=unattended session and finds no judge work until teamai score --drain runs.
• Attended auto publishing runs in the detached pass that judges finished sessions (spawnDetachedScoreSessions, src/hook-handlers.ts). Covered by a test with publish: { attended: auto }.
• With a publish or scoring policy other than off, teamai doctor names each enabled tool teamai reads no sessions from. Nothing for those tools is silently skipped.
• The guide states the limitations: runs, judge attempts and leases live in the runner's data home, and a lapsed session lease or a failed judge attempt is final. A runner killed before its drain loses that judging. A drain run from inside a session judges and publishes it as it stands.

Definition of done

• docs/guide/scorers.md (the policies and defaults, which setting governs passes and which sessions, score --drain and the rule that every unattended driver ends with it, the two places it runs (the job's last step sees the session complete; a drain from inside a session sees it as it stands, and its later turns are not judged or published again), that a failed judgment is final, daily_limit per data home next to the unattended: auto default), docs/guide/dashboard.md and docs/guide/workflow-advanced.md (where session push and the archive are documented; the drain at the end of an unattended run), docs/guide/sharing.md (sessions.publish) and their zh-CN pages. The team file and member override examples above go in these guide pages, and the behavior change with its opt-out comes first.
• CHANGELOG.md under ## [Unreleased]: the two entries above, in ### 💥 Breaking Changes and ### Changed.
• docs/designs/session-user-repo-sync.md: the archive is on teamai-reports.
• skill-data/setup/references/setup-admin.md (the line that says reports and learnings use teamai-reports and teamai-learnings): pushed sessions also live on teamai-reports.
• skill-data/workflow/SKILL.md: the one drain line above. No other skill text changes.
• session push, score and workflow start help text covers --session on push, --drain, and that an unattended local run drains at the end; skill-data/core/references/commands.md regenerated with npx vitest run commands-reference -u.
• Real-CLI record: one unattended workflow start --driver local run with no settings, showing judging, labels and archived sessions; the same run with unattended: off, showing none; one teamai score --drain with nothing left and one with a publish failure (exit 1); one session push --session on a git-kind team landing on teamai-reports; one session resume of a session archived before the change.

Out of scope

• Offering ask to a person ([feat] Default workflows a team can override, starting with session-close #1388).
• Moving archives that already exist on the default branch. They stay readable there.
• Recovering judge work from a runner killed before its drain.
• Reading sessions of tools teamai has no session adapter for.
• Answering approvals. Unattended mode answers no approval of the local driver and never turns --auto-approve on. A CI job passes --auto-approve as today for optional approvals; without it they are answered no and their steps are skipped. A required approval fails its step with hitl_required, as it does in CI today (docs/guide/workflow.md, "Start from CI"), and a person retries it from a terminal with teamai workflow retry.

Notes

• Why unattended defaults to auto: nobody is there to answer ask, and an automation's sessions are the team's to review. A team that does not want that opts out with unattended: off.
• Capture applies only to tools with a session adapter (src/session-flow/adapters/platform-facts.ts). Pi, Muse, Copilot and others are not captured today, so no policy reaches their sessions.
• The skill driver handles unattended runs itself (feat(workflow): add session discovery and managed public leases #1304): --describe starts with --auto-approve without asking, and the run's own pass judging keeps running through done.judging. This issue adds the one drain line for what that loop does not cover (sessions and publishing). Nothing routes to the local driver. resolveRunMode comes from feat(workflow): add session discovery and managed public leases #1304, through [feat] Run mode: tell attended runs from unattended ones #1384, which depends on [feat] Add the session skill driver and invocable workflows #1086.
• Unattended mode answers no approval of the local driver: with no HITL channel, answerWithoutHitl (src/workflow/engine.ts) answers optional approvals no unless --auto-approve is given, and fails a required one with hitl_required. The skill driver's unattended --describe starts with --auto-approve because nobody is there to pick (feat(workflow): add session discovery and managed public leases #1304).
• Drain loop to reuse: scoreSessions (src/score-sessions.ts:43-68) already takes leases with takeJudgeLease and judges them with judgeLease, over localJudgeOffers() (:29). The drain runs that loop over passes and sessions, without its 30-minute deadline, then publishes. Label publishing: what score --push does today (src/index.ts, score command). Usage: reportUsageToTeam (src/team-push.ts:933), called by pull at src/pull.ts:3340.
• Entries left for other drivers: takeJudgeLease skips an entry whose scorer's model none of the driver's tools runs and leaves it open (judge_model_unavailable, SkippedEntry in src/workflow/scoring.ts). The judge tool does not depend on the tool that wrote the transcript, so the drain cannot name a tool a session needs.
• Step sessions of a finished run: each settled pass reports its tool and session (src/workflow/step-state.ts:66-71). archiveSessions takes session chains (src/session-flow/archive.ts:129-133); its only caller today builds them from unsettled steps (src/workflow/handoff.ts:293-308).
• Archive location, one switch: sessionArchiveRoot (src/session-flow/session-cmd-shared.ts:213-217). Self mode already returns the reports checkout there. Writers then go through updateReports as self mode does (resolveRepoRoot, :226-250). Readers: openSessionArchive (src/session-flow/archive.ts:50-120, which today fetches the default branch through fetchLatest at :105-120) and alreadyArchived (:191-205). Run handoff writer: archiveSessions (:129-189). Archive writer for session migrate: src/session-flow/session-migrate-cmd.ts:441, :491. session pull rebuilds its indexes in src/session-flow/session-cmd.ts:309-330.
• No new branch: pull runs git pull --ff-only on the full team clone (src/utils/git.ts:522-527), which fetches every head, so a separate branch would save no fetch.
• Finished sessions: FINISHED_AFTER_MS (src/session-flow/session-work.ts:138), applied at :265. Stop is registered for every tool; SessionEnd only for Copilot (src/builtin-hooks.ts:336-350).
• Judge attempts: a session is judged once, so a released, lapsed or failed session attempt never opens it again (sessionEntryState, src/scorers/entry.ts:87-103). A pass entry closes as failed on its first failed attempt (entryState, :59-80). Runs live in <dataHome>/workflows/runs (runsDir, src/workflow/store.ts), attempts in <dataHome>/scores (src/scorers/attempts.ts:1-5, :120-122).
• daily_limit is counted per data home (src/scorers/settings.ts:6-7), so on a fresh runner it caps one job, not one day.
• Judge work: takeJudgeLease, the request JudgeRequest (its entries field says "Never a session"), and the idle answer nextJudgeLease, all in src/workflow/scoring.ts. The drain takes every judge entry in the data home, including earlier runs' on a long-lived machine. feat(workflow): add session discovery and managed public leases #1304 added a run filter for the skill driver; the drain does not need it.
• The local driver returns at done in driveRun (src/workflow/drivers/local/local-driver.ts); the unattended drain goes there.
• Today's background judge: src/hook-handlers.ts:162-169 (spawnDetachedScoreSessions), src/score-sessions.ts:43-68.
• Scoring settings: src/scorers/settings.ts:27-35 (keys), :109-130 (sessionScoring). Member keys: src/types.ts:900-907. Team config: src/types.ts:664-738.
• Labels and stats share the usageReport opt-out (isRecordingEnabled, src/workflow/records.ts; src/scorers/label-store.ts:121); unchanged.
• Lease ref format: s<13 hex>/a<8 hex> (LEASE_REF, src/scorers/entry.ts:20), for example s5d1e07c2a9b40/a91c0d7e0.
• Unattended sessions become eligible only when a drain reaches them. A session no drain ever reaches stays unjudged and unpublished, under the documented runner-loss limitation. A long-lived VM runs teamai score --drain at the end of each of its jobs.