Skip to content

[feat] Record what a session, step or run delivered #1387

Closed

Description

@SaulMoro

Example

A learning names the session it came from:

$ teamai contribute --file notes.md --title "Pod startup timeouts" --session-id 7c1e…
---
title: Pod startup timeouts
author: alice
date: 2026-10-24
tags: [k8s, startup, troubleshooting]
session: { platform: claude-code, sessionId: 7c1e… }
---

A learning's id is its path under learnings/ without .md:

learnings/pod-startup-timeouts-2026-10-24-k3x9q2.md       → pod-startup-timeouts-2026-10-24-k3x9q2
learnings/k8s/pod-startup-timeouts-2026-10-24-k3x9q2.md   → k8s/pod-startup-timeouts-2026-10-24-k3x9q2

A session's outcomes stay on this machine, as dashboard events:

{"type":"session_outcome","sessionId":"7c1e…","tool":"claude","runMode":"attended",
 "outcomes":[{"kind":"commit","sha":"9f2c…"},{"kind":"pr","url":"https://github.com/acme/app/pull/42"},
             {"kind":"learning","id":"pod-startup-timeouts-2026-10-24-k3x9q2"}]}

A step's outcomes come from what the run already records, plus what the step's own commands report. Inside a workflow attempt, contribute appends to the attempt's outcome file:

$ echo "$TEAMAI_WORKFLOW_ATTEMPT"
r3f9…/a2
$ teamai contribute --file "$FILE" --title "Retry backoff"
$ cat <dataHome>/workflows/runs/r3f9…/outcomes/a2.jsonl
{"kind":"learning","id":"retry-backoff-2026-10-24-p0x1za"}

The run keeps prs; its union is derived:

// workflows/runs/r3f9….json on teamai-reports, end record
{ "run": "r3f9…", "runMode": "unattended", "prs": ["https://github.com/acme/app/pull/43"],
  "steps": [ { "step": "fix", "pass": 1, "outcomes": [ { "kind": "pr", "url": "https://github.com/acme/app/pull/43" } ] },
             { "step": "close", "pass": 1, "outcomes": [ { "kind": "learning", "id": "retry-backoff-2026-10-24-p0x1za" } ] } ] }

The usage report and digest carry counts only, per run mode and outcome kind:

runMode       outcome    sessions   tokens/session   interventions/session
attended      pr         12         410k             6.1
attended      learning   5          220k             3.0
attended      none       31         95k              2.2
unattended    pr         20         380k             0.4
not recorded  none       7          120k             1.9
# stats/alice.yaml on teamai-reports, beside today's counters
outcomes:
  attended:
    pr:       { sessions: 12, tokens: { input: …, output: … }, interventions: 73 }
    learning: { sessions: 5, tokens: { input: …, output: … }, interventions: 15 }
    none:     { sessions: 31, tokens: { input: …, output: … }, interventions: 68 }
  unattended:
    pr:       { sessions: 20, tokens: { input: …, output: … }, interventions: 8 }
  notRecorded:
    none:     { sessions: 7, tokens: { input: …, output: … }, interventions: 13 }

A session can be in several rows (a PR and a learning), so the rows overlap and their totals are not additive.

Behavior

Kind Session Step pass Run
commit commits authored by the member's git identity on the session's checkout, in gitAtStart.head..<gitHead of its last Stop event>, once the session is finished (#1385) commits on the step's branch derived: union of steps
pr a PR open for the checkout's branch that contains one of the session's commits. Looked up once, when the usage report folds a finished session, and cached. A session with no commit gets none the run's delivery receipt for that step and pass (deliver: pr) prs, as today
learning contribute for the session --session-id names (else the tool's session variable): the learning's id. Also inside a workflow attempt: the session it names is not the attempt's contribute run inside the attempt (TEAMAI_WORKFLOW_ATTEMPT set): appended to the attempt's outcome file, put on the pass when the attempt settles derived: union of steps
Data Where it lives Leaves the machine
Session outcomes with ids, shas and URLs local dashboard events no
A step's reported outcomes <dataHome>/workflows/runs/<run>/outcomes/<attempt>.jsonl, then the pass the pass's outcomes, with the run record
PR lookups <dataHome>/dashboard/outcome-prs.json, by session no
Counts per (runMode, outcome kind): sessions, tokens, interventions stats/<member>.yaml, under outcomes yes, as today's counts do
Step outcomes the run's end record yes, as run records do today
A learning's session its frontmatter on teamai-learnings yes, with the learning

A session, step or run can have several outcomes, or none. none is the empty list, shown as its own row.

Acceptance criteria

  • teamai contribute writes session: { platform, sessionId } into the learning's frontmatter before the learning is queued, for the session it already resolves (--session-id, else the tool's session variable). The platform is resolved as score --session resolves it: the session variable's tool family, else the session's captures.
  • A learning's id is its path under learnings/ without .md, so a namespaced learning's id carries its namespace (k8s/…). A root learning's id is its file name without .md.
  • Session outcomes are dashboard events, folded with the session's other events. They are never written into captures. A session with no capture (a tool teamai has no session adapter for) still gets its outcomes, so the per-mode report counts it.
  • The Stop dashboard event records gitHead, so a session's commit range is known without reading git at fold time.
  • With TEAMAI_WORKFLOW_ATTEMPT set (feat(workflow): add session discovery and managed public leases #1304), contribute appends {"kind":"learning","id":…} to <dataHome>/workflows/runs/<run>/outcomes/<attempt>.jsonl. Both drivers read that file when the attempt settles and put its entries on the pass's outcomes. The step's result file and workflow report are unchanged. Covered by a test per driver.
  • Step outcomes also come from run.deliveries (by step and pass). A run record gains no second PR field next to prs.
  • Outcome rows count the same sessions [feat] Run mode: tell attended runs from unattended ones #1384 counts as the member's: sessions with launchedBy are left out, and step outcomes live on run records only. launchedBy is read for the session an outcome belongs to, not for the process that writes it.
  • The usage report has counts only, under outcomes: { <runMode | notRecorded>: { <kind | none>: { sessions, tokens: { input, output }, interventions } } }. An attended session is added once it is finished (6 hours idle). An unattended session is added when teamai score --drain reports usage at the end of its job ([feat] Publish and score sessions by run mode #1385), since the drain is where unattended sessions finish, so an ephemeral runner's sessions are counted. A per-session reported flag in a new baseline file, reported-outcomes.json beside today's baselines, keeps it from being added twice. No session id, commit sha or PR URL is published.
  • Outcome rows use [feat] Run mode: tell attended runs from unattended ones #1384's runMode; records without it are not recorded. This issue adds the outcome dimension and does not re-split the existing metrics.
  • The PR lookup uses providerForRemote for the checkout's origin, with the credentials deliver: pr uses, and is skipped without them. A lookup that fails leaves pr out; it never fails the hook or the report.

Definition of done

  • docs/guide/reporting.md, docs/guide/dashboard.md and their zh-CN pages, with the examples above. docs/guide/knowledge.md and its zh-CN page: one line on the session field.
  • skill-data/share/references/doc-template.md: the session field in the frontmatter table (the share workflow writes learnings).
  • contribute help text: --session-id now also records the session in the learning, and inside a workflow attempt the learning is recorded on the step. skill-data/core/references/commands.md regenerated with npx vitest run commands-reference -u.
  • Real-CLI record: one session with a commit and a learning, and the usage report and digest that show them; one workflow step that runs contribute, and its pass's outcomes.

Out of scope

  • Issues and specs as outcome kinds (later).
  • Cost in currency; tokens only, as today.
  • A pointer from a learning to the archived copy of its session. Readers find it from session with findArchived.

Notes

  • contribute: src/contribute.ts:141-275. It queues the content at :228 and uses the session id only for markContributed at :268-270. File names: generateFilename (:120-130); the namespaced path is built at :210. Frontmatter helpers: parseFrontmatter / stringifyFrontmatter (src/utils/frontmatter.ts:73, :93), as src/maintenance/confidence.ts:96-139 already uses them.
  • Platform resolution for a session id: reuse sessionOf's pattern (src/session-flow/scoring.ts:57-76). agentSessionFromEnv (src/utils/session-id.ts:81) gives the id and the tool family, and AgentSession.platformForTool maps the family to a session platform.
  • Session subject shape: SessionSubjectSchema (src/scorers/label.ts:13).
  • Start head: addGitStateAtStart (src/dashboard-collector.ts:2500-2504) puts gitAtStart on session_start. Folding: aggregateSessionMetrics (src/dashboard-collector.ts:2082).
  • Captures describe a session's start and are swept a day after the session file is gone (src/session-flow/captures.ts:204-217), so they are not the place for outcomes. They exist only for tools with a session adapter (src/session-flow/adapters/platform-facts.ts); Pi, Muse, Copilot and others are not captured today.
  • Run records: buildEndRecord (src/workflow/records.ts) takes prs from run.deliveries. Receipts carry step and pass (DeliveryReceiptSchema, src/workflow/delivery-schema.ts).
  • Why an outcome file and not the step's result: the local driver's prompt tool writes its whole result file at the end (TEAMAI_WORKFLOW_RESULT, set in createLocalToolAdapter, src/workflow/drivers/local/local-tool-adapter.ts), which would overwrite a line contribute added, and command steps and skill-driven steps do not use that file. The attempt marker reaches every process of an attempt in both drivers (attemptEnv, src/workflow/attempt-marker.ts, feat(workflow): add session discovery and managed public leases #1304).
  • PR provider: providerForRemote (src/providers/registry.ts:234), as deliver: pr picks it (originOf, src/workflow/pr-delivery.ts).
  • Usage aggregation: src/team-push.ts:933-985, baselines reportedBaselines (:817). stats/<user>.yaml is counts only by design (src/types.ts:1743-1771).
  • The implementer decides whether session outcomes are one session_outcome event or fields on the session's last event, and what contribute does when neither the session variable nor a capture names the platform.

Activity

  1. SaulMoro commented on Oct 25, 2026

    @SaulMoro
    CollaboratorAuthor

    Rewrote the body. The usage report carries counts only, per run mode and outcome kind, because stats/<user>.yaml is counts only by design (src/types.ts:1726-1754). Ids, shas and URLs stay local as dashboard events.
    Those events also cover sessions with no capture, so a tool teamai has no session adapter for still counts.
    The session subject is spelled { platform, sessionId }, as labels spell it (src/scorers/label.ts:13), and the frontmatter archive field is dropped.
    Step outcomes come from run.deliveries next to prs, and a session's pr outcome needs one of its own commits. The split by mode stays with #1384.

  2. SaulMoro commented on Oct 25, 2026

    @SaulMoro
    CollaboratorAuthor
    • A step's learning now goes through a per-attempt file, <dataHome>/workflows/runs/<run>/outcomes/<attempt>.jsonl, which contribute appends to when TEAMAI_WORKFLOW_ATTEMPT is set. The tool's result file would overwrite it, and command steps and skill-driven steps do not use that file.
    • A learning id is its path under learnings/ without .md, so a namespaced learning keeps its namespace.
    • Named the stats shape and its per-session baseline, the Stop event's gitHead for the commit range, and where PR lookups are cached. Outcome rows count only the member's own sessions.
  3. SaulMoro commented on Oct 25, 2026

    @SaulMoro
    CollaboratorAuthor

    Said when an unattended session's outcomes are counted. That happens when teamai score --drain reports usage at the end of its job (#1385), because the drain is where unattended sessions finish. Before, nothing counted them on a runner that never runs pull. Attended sessions are still counted at 6 hours idle.
    Updated the UserStats line numbers, which moved on main.

  4. SaulMoro commented on Oct 26, 2026

    @SaulMoro
    CollaboratorAuthor

    session-close (#1388) runs contribute --session-id inside a workflow attempt. The learning row now says the session outcome goes to the session --session-id names, even inside an attempt, and that launchedBy is read for the session an outcome belongs to, not for the process that writes it. Without this, attended learnings would drop out of the report once #1388 lands.

  5. 16 remaining items

  6. SaulMoro commented on Oct 27, 2026

    @SaulMoro
    CollaboratorAuthor

    This was generated by AI during triage.

    Status: #1471 is merged (bd3adf9). contribute writes session: { platform, sessionId } into the learning and records the learning as an outcome of that session. Inside a workflow attempt it also appends to the attempt's outcome file, which settles onto the pass with the step's deliveries. Stop events carry gitHead and gitBranch. The usage report folds each finished session once (reported-outcomes.json) into outcomes.<runMode | notRecorded>.<kind | none>, and digest, the dashboard and the guides show it.

    Left: counting an unattended session when teamai score --drain reports usage at the end of its job. #1478 (59b2c07) runs the usage report from the drain, but the outcome fold still finishes a session only after 6 hours idle (isFinished in src/session-outcomes.ts) and does not read the drain's mark. A runner gone before then never reports those outcomes, as docs/guide/reporting.md now says. The fix also changes that sentence and its zh-CN page, and needs a real-CLI drain that shows an unattended session's outcome in stats/<member>.yaml.

    This stays open until that lands.

  7. added a commit that references this issue on Oct 27, 2026
    ccb16d2
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    enhancementNew feature or request

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions