[feat] session-to-task: turn a session's bug fix into a benchmark task #1402

Example

person: turn this fix into a benchmark task
agent:  teamai workflow start --describe session-to-task --driver skill
        proposes {"session":"7c1e…","source":"claude-code","suite":"benchmarks/frontend",
                  "start":"3f2c9a7d1e0b4c6a8f5d2e1b0c9a8f7e6d5c4b3a","fix":"9a1d4e7c2b8f0a3d6e5c1b9f8a7d2e4c6b0a1f3e"}
person: confirms, picking "Same, but ask me at optional approvals"
agent:  works the draft step with the person, then asks the approval
        "Propose benchmarks/frontend/tasks/pod-startup.yaml to the team as a pull request?"
person: yes
step:   teamai push --file /tmp/pod-startup.yaml --path benchmarks/frontend/tasks/pod-startup.yaml
        ✔ Pull Request created: https://github.com/acme/team/pull/77

A member fixed a bug with their agent and keeps it as a benchmark task. The session that fixed the bug drafts the task with them, so no transcript has to be read back. For a suite in a namespace, suite is its directory, such as benchmarks/web/frontend, and the task goes to benchmarks/web/frontend/tasks/pod-startup.yaml.

The task file it proposes, in #1180's format:

# benchmarks/frontend/tasks/pod-startup.yaml
schemaVersion: v1alpha1
title: Pods time out on first start
prompt: |
  The first start of a pod times out after 30 s on a cold node. Make it start.
startingCommit: 3f2c9a7d1e0b4c6a8f5d2e1b0c9a8f7e6d5c4b3a
check:
  run: npx vitest run test/startup.test.ts
  protect: ['test/**', 'vitest.config.ts', 'package.json', 'package-lock.json']
  add: ['test/startup.test.ts']        # files under protect the fix added or changed: read from source.fix, never in the agent's clone
source: { platform: claude-code, sessionId: 7c1e…, fix: 9a1d4e7c2b8f0a3d6e5c1b9f8a7d2e4c6b0a1f3e }

The fix brought its own test. A trial's agent must not see it, or it could fit its change to the expected result instead of fixing the bug. check.add (#1180) keeps it out of the agent's clone and writes it in just before the check:

step                          test/startup.test.ts
trial: clone at 3f2c9a7       absent: the clone holds only 3f2c9a7 and its history, so not even git has the fix
trial: check                  protect restored, then written from source.fix   → measures the agent's change against the hidden test
--try at startingCommit       written from source.fix                          → must fail: with the test present, that is its answer about the bug
--try at source.fix           protect restored, then written from source.fix   → must pass

The team repo's CI tries the task's check on the pull request with #1180's --try:

# .github/workflows/benchmark-tasks.yml in the team repo
on:
  pull_request:
    paths: ['benchmarks/frontend/tasks/*.yaml']
jobs:
  try:
    runs-on: ubuntu-latest
    timeout-minutes: 30
    steps:
      - uses: actions/checkout@v5          # the team repo, at the pull request
        with: { path: team, fetch-depth: 0 }
      - uses: actions/checkout@v5          # the suite's repo, with every commit a task names
        with: { repository: acme/frontend, path: project, fetch-depth: 0, token: "${{ secrets.PROJECT_READ_TOKEN }}" }
      - uses: actions/setup-node@v4
        with: { node-version: 22 }
      - run: npm install -g teamai-cli
      - working-directory: project
        shell: bash
        run: |
          git -C ../team diff --name-only --diff-filter=AM "origin/${{ github.base_ref }}...HEAD" -- 'benchmarks/frontend/tasks/*.yaml' |
          while read -r file; do
            suite=$(basename "$(dirname "$(dirname "$file")")")
            teamai workflow start "$suite" --benchmark --try "../team/$file" --driver local --sandbox
          done

task           at startingCommit      at source.fix (protect restored)   job
pod-startup    check fails  ✔         check passes  ✔                    passes: its test is hidden in check.add and written in before each check
old-bug        check passes ✖         not tried                          fails: "check passes without changes at 3f2c9a7"
bad-fix        check fails  ✔         check fails   ✖                    fails: "check fails at the fix 9a1d4e7"
not-added      check fails  ✔         check fails   ✖                    fails: "check fails at the fix 9a1d4e7": the fix's test is not in check.add, so neither commit has it
add-missing    not tried              not tried                          fails: "test/startup.test.ts is not a file at the fix 9a1d4e7"
no-fix         check fails  ✔         not tried                          passes, and says the check was not tried on a fix
criteria-only  no check               no check                           passes, and says the task has no check to try
missing-tool   could not run          not tried                          fails, naming the command and 3f2c9a7
setup-error    setup failed           not tried                          fails, naming setup and 3f2c9a7

With a check that runs the whole suite, not-added fails earlier: "check passes without changes at 3f2c9a7". --try sees only the check's exit, not why it failed.

The job covers one project: its repository and path filter name the same suite repo. A team with suites for several repositories adds a job per project with matching filters. PROJECT_READ_TOKEN reads that project. The team marks the job required in its merge rules, so a task that fails it never reaches the suite. The --try messages are #1180's.

Behavior

The default:

schemaVersion: v1alpha1
name: session-to-task
description: Turn the bug this session fixed into a benchmark task, proposed to the team as a pull request. Use when the person asks to keep this fix as a benchmark task or add it to a benchmark suite.
invoke: ask
inputs: { session: string, source: string, suite: string, start: string, fix: string }
steps:
  - id: draft
    interactive: true
    prompt: |
      Turn the bug this {{ inputs.source }} session {{ inputs.session }} fixed into one task of the benchmark suite in {{ inputs.suite }}.
      Work it with the person. The task starts at commit {{ inputs.start }}; the fix is commit {{ inputs.fix }} (empty when there is none).
      Write the task file outside the repository, in the format of the files under {{ inputs.suite }}/tasks/:
      schemaVersion v1alpha1; title; prompt, the bug as the person first described it, without the fix;
      startingCommit {{ inputs.start }}; exactly one of check or successCriteria;
      and source with platform {{ inputs.source }}, sessionId {{ inputs.session }} and, when there is a fix, fix {{ inputs.fix }}.
      check has run, and protect listing every test, config, script and manifest the check reads.
      When there is a fix, check also has add: every file under protect that the fix added or changed and the check reads
      (tests, fixtures, helpers; git diff --name-only {{ inputs.start }} {{ inputs.fix }} lists the candidates), one path each, no globs.
      The trial's agent never sees the add files, so it cannot fit its change to them.
      Every add path must be a file at commit {{ inputs.fix }}; if one is not, tell the person to commit it and start session-to-task again, and return skip.
      When an add file calls a function, file, option or export that commit {{ inputs.start }} lacks, name it and its signature in prompt,
      so the task can be solved without seeing the test.
      If the person decides the session is not a good task, return {"verdict":"skip","file":"","path":""}.
      Otherwise return verdict draft, the file's path, and {{ inputs.suite }}/tasks/<task id>.yaml as path,
      where the task id is letters, digits, "-" or "_".
    output: { verdict: draft | skip, file: string, path: string }
  - id: propose
    needs: [draft]
    if: steps.draft.outputs.verdict == 'draft'
    approve: Propose {{ steps.draft.outputs.path }} to the team as a pull request?
    command: teamai push --file "$FILE" --path "$TASK_PATH"
    env: { FILE: "{{ steps.draft.outputs.file }}", TASK_PATH: "{{ steps.draft.outputs.path }}" }

session-to-task is a teamai default workflow, served and overridden like session-close (#1388): a team's workflows/session-to-task.yaml replaces it. Nothing offers it automatically. A person asks their agent for it, or the generated workflow-session-to-task skill points the agent at it.

What describe proposes, as a value and a why on each entry of describe's inputs (#1388's seam):

start     fix (HEAD)   proposal
3f2c9a7   9a1d4e7      both, as is
3f2c9a7   3f2c9a7      start only: HEAD is still the starting commit
3f2c9a7   9a1d4e7 *    start only: tracked files modified; commit the fix, then describe again
(none)    9a1d4e7      neither: the person gives both

A proposal is always a full 40-character commit. Each of these inputs carries a why, also when it has no value, so the person sees where a commit came from or why it is missing:

{"name":"start","type":{"type":"string"},"value":"3f2c9a7d…","why":"The commit this session started at."},
{"name":"fix","type":{"type":"string"},"why":"No value: tracked files are modified. Commit the fix, then ask again, or give the commit."},
{"name":"suite","type":{"type":"string"},"value":"benchmarks/frontend","why":"The one benchmark suite whose repo is this checkout's origin."}

session and source carry a why like #1388's, worded for this workflow: "This session: the one that fixed the bug." The other fix and suite reasons follow the "No value when" column: HEAD is still start; no starting commit; no suite, or several, match origin (naming them).

teamai push --file <local file> --path <team path>:

$ teamai push --file /tmp/pod-startup.yaml --path benchmarks/frontend/tasks/pod-startup.yaml
✔ Pull Request created: https://github.com/acme/team/pull/77

$ teamai push --file /tmp/pod-startup.yaml --path benchmarks/frontend/pod-startup.yaml
✖ benchmarks/frontend/pod-startup.yaml is not a benchmark task path. A task goes in benchmarks/<suite>/tasks/<task>.yaml or benchmarks/<namespace>/<suite>/tasks/<task>.yaml.

$ teamai push --file /tmp/pod-startup.yaml --path benchmarks/missing/tasks/pod-startup.yaml
✖ benchmarks/missing/suite.yaml does not exist, so benchmarks/missing/tasks/pod-startup.yaml belongs to no suite. Pick a suite from teamai list benchmarks.

$ teamai push --file /tmp/pod-startup.yaml --path benchmarks/frontend/tasks/pod-startup.yaml --role web
✖ --path names the destination, so it cannot be combined with --skill, --role, --project or --shared.

Acceptance criteria

• The default parses with parseWorkflow, with invoke: ask, interactive: true on draft, the output { verdict: draft | skip, file, path }, and propose guarded by if: steps.draft.outputs.verdict == 'draft' with an optional approve. It is served behind loadCachedWorkflow and listCachedWorkflows like session-close ([feat] Default workflows a team can override, starting with session-close #1388), and a team file workflows/session-to-task.yaml replaces it.
• A run whose draft returns {"verdict":"skip","file":"","path":""} succeeds, asks no approval and opens no pull request. Covered by a test.
• describe proposes start, fix and suite with the rules above, each with a why, also when it proposes no value. Covered by tests: an earliest session_start with gitAtStart; one whose gitAtStart.dirty is true (no start); an earliest one without it and a later resume with it (no start); HEAD equal to start; a dirty checkout; one matching suite; two matching suites; an SSH origin matching an HTTPS repo; a namespaced suite proposed as benchmarks/web/frontend; no agent session (no values).
• teamai push --file --path puts only the destination in the pull request. Covered by a test whose team clone also has a modified teamai.yaml and another changed resource: the pull request's tree differs from its base only at the destination. A skills/ destination adds or rewrites no other skill file. In self mode the pull request holds .teamai/<path>, and the active checkout's files, index and HEAD are unchanged.
• The error rows of the two push tables are covered by tests, including a structurally wrong task path and a missing suite.yaml as separate cases.
• --path takes a path under any placeable type's root (skills/, scorers/, benchmarks/, …), validated against the namespaces push resolves today and never relocated. A path that escapes the team root, or goes through a symlink, is refused. Covered by a scorers/ test and a ../ test.
• The guide's CI job, run on a fixture team repo pull request with no teamai init, passes for a task whose check fails at startingCommit and passes at its fix; passes for a task whose test the fix added and check.add lists; passes for a task whose fix added a case to an existing test file that check.add lists; fails for the first task without check.add; fails naming the path for an add path that is not a file at the fix; and fails with [feat] Run benchmark suites as workflows #1180's message for one whose check passes at startingCommit.

Definition of done

• docs/guide/benchmarks.md and docs/guide/zh-CN/benchmarks.md: a section "Turn a session into a task" with the session example, the task file, the CI job and its outcome table.
• docs/guide/workflow.md and its zh-CN page: session-to-task in the list of default workflows ([feat] Default workflows a team can override, starting with session-close #1388), with its inputs.
• docs/guide/sharing.md and its zh-CN page: push --file --path, with the flag table.
• push help text for --file and --path; skill-data/core/references/commands.md regenerated with npx vitest run commands-reference -u.
• CHANGELOG.md under ## [Unreleased] › ### Added: the session-to-task default workflow and teamai push --file --path.
• Real-CLI record: one attended session that runs session-to-task to a pull request on a fixture team repo, and one run of the CI job's --try command on that pull request's branch.

Out of scope

• Trying the check before the pull request opens, on the member's machine. The check runs project setup and code, so it runs in the team's CI sandbox, where [feat] Run benchmark suites as workflows #1180 already runs trials.
• --try, source.fix and check.add themselves: [feat] Run benchmark suites as workflows #1180 adds them, to the contract in Notes.
• Turning a session another member archived into a task. The draft step needs the session that fixed the bug; a transcript reader is a separate issue.
• Offering session-to-task automatically, for example from the friction reminder. [feat] Default workflows a team can override, starting with session-close #1388's reminder offers session-close only.
• A results page for tasks or suites.

Notes

• This issue replaces the "Second slice: session-to-task" that [feat] Default workflows a team can override, starting with session-close #1388 carried, with its examples and acceptance. From [feat] Default workflows a team can override, starting with session-close #1388 it reuses only default-workflow resolution and the describe value and why seam.
• From [feat] Run benchmark suites as workflows #1180 (PR feat(workflow): run benchmark suites as workflows #1379), needed here. [feat] Run benchmark suites as workflows #1180 carries this contract and the two --try messages verbatim ("check passes without changes at ", "check fails at the fix "):
  • check.add, a list of repo files that requires source.fix: one path each with no glob, each matched by a protect entry as git matches :(glob) pathspecs. Their content comes from source.fix, whether the starting commit has the file or not. The bytes are read from source.fix in the first of the trial's sources that has it (the run's checkout, then the commit cache) into an index of its own, as restoreProtected reads startingCommit (readStarting, src/workflow/benchmark/trial.ts). The fix never enters the agent's clone: freshClone fetches only startingCommit and its ancestors, so the clone has no copy of fix to read and none for the agent to find, and git checkout <fix> -- <paths> in the clone fails. A trial's check runs restoreProtected, writes the add paths from that index, then checks them byte for byte as restoreProtected checks the protected paths. The baseline writes them the same way after freshClone and before setup. start makes sure source.fix is reachable for a task with check.add (unreachableTask, src/workflow/benchmark/start.ts); --try checks it for any task that sets it (firstUnreachable, same file), and the check and baseline terms (src/workflow/benchmark/terms.ts) carry add and fix. An add path that is not a file at fix fails the check trial_prepare_failed, naming the path and the commit. Pull and review --branch refuse a glob, a path no protect entry matches, or add without source.fix. Same .strict() reason: first release.
  • source.fix, an optional full commit in the task file's source (TaskFileSchema, src/workflow/benchmark/suite.ts). The schema is .strict(), so a teamai without the field rejects a file that has it; it ships with [feat] Run benchmark suites as workflows #1180's first release.
  • teamai workflow start <suite> --benchmark --try <task file>: reads the task file and the suite.yaml two directories up from it, not the pulled cache, and needs no teamai init; the leaf name checks that suite's name. Like [feat] Run benchmark suites as workflows #1180's CI job it runs in a checkout of the suite's repo and clones from it (freshClone, src/workflow/benchmark/trial.ts). It makes sure startingCommit and, when set, source.fix are reachable (firstUnreachable: the checkout, else a fetch by SHA from the suite's repo into a temporary cache). An add path that is not a file at source.fix exits non-zero naming the path and the commit, before setup. It runs setup and the check at startingCommit with the check.add files written from source.fix, where the check must fail by its own exit, and, with source.fix, setup and the check at fix after restoreProtected puts the protect paths back as startingCommit has them and the check.add files are written again, where it must pass. A setup failure or a check classified could_not_run (checkVerdict, src/workflow/benchmark/check.ts) at either commit exits non-zero naming the commit and cause. With successCriteria it validates the files, runs nothing, exits 0 and says there is no check to try. No arms, trials, judges or record. Needs --sandbox like trials.
• Why the check is not tried inside the workflow: every command step carries the attempt marker (attemptEnv, defined in src/workflow/attempt-marker.ts, passed by the local driver in src/workflow/drivers/local/local-driver.ts and by the skill runner in src/workflow/drivers/skill/skill-runner.ts), so a nested teamai workflow start … --benchmark is refused nested_workflow (nestedStart, src/workflow/attempt-marker.ts). The fresh clone at a commit and the protect restore exist only as trial terms of [feat] Run benchmark suites as workflows #1180's generated plan (compilePlan, src/workflow/benchmark/plan.ts), which a YAML step cannot declare.
• Interactive steps: worked by the session's own agent with the person, exempt from the read-only check (docs/guide/workflow.md, "interactive"). session-close's learning step is the precedent ([feat] Default workflows a team can override, starting with session-close #1388).
• describe's input seam: inputsOf (src/workflow/drivers/skill/preflight.ts, feat(workflow): add session discovery and managed public leases #1304), extended with value and why by [feat] Default workflows a team can override, starting with session-close #1388, keyed by the resolved workflow's name; this issue adds the session-to-task entry beside session-close's. The caller's session: agentSessionFromEnv (src/utils/session-id.ts); source from its family through platformForTool (src/session-flow/agent-session.ts).
• start: dashboard events, read with readEvents (src/dashboard-collector.ts). Take the calling session's earliest session_start, then its optional gitAtStart.head (HeadState is { head, dirty }, src/utils/git.ts), written by addGitStateAtStart (src/dashboard-collector.ts). Resume and compaction also write a session_start, so "earliest" matters. collectSession (src/session-collector.ts) picks the earliest start that has git state, which is a different rule; do not reuse it unchanged. Session captures (src/session-flow/captures.ts, agent.project.commit) are a different store.
• fix: getHeadState(cwd) (src/utils/git.ts). suite: getRemoteUrl(cwd) and remotesMatch (src/utils/git.ts) against each resolved cached suite's repo. feat(workflow): run benchmark suites as workflows #1379 exposes loadCachedSuite (src/workflow/cache.ts) but no enumeration; add a listCachedSuites beside it, resolving namespaces as loadCachedSuite does, and propose the directory of its relativePath.
• push --file --path is an exact-file route inside push. Copy the source bytes to the destination and keep the source. Reuse push's scope, lock, self-mode worktree (withKnowledgeWorktree, src/utils/reports-branch.ts), provider and pull request code (createPrWithFallback, src/push.ts). Bypass scanner selection, resource-handler transformations (the skill handler rewrites frontmatter and CONTRIBUTORS, src/resources/skills.ts), namespace relocation, sweeper staging and pending teamai.yaml inclusion (src/push.ts). Stage only the destination. Check containment with the existing path-safety helpers; pushRepoResourceItem (src/resources/repo-resource.ts) does not. --skill <path> and resolveRequestedSkill (src/push.ts) are the precedent for naming one resource. benchmarks is a placeable type from [feat] Run benchmark suites as workflows #1180 (src/push-namespaces.ts).