Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
2 changes: 1 addition & 1 deletion docs/guide/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ The run is stored in this scope's TeamAI data directory. The command exits when

With [usage reporting](reporting.md) on, every run publishes a privacy-preserving record to the team's `teamai-reports` branch: once when it starts and once when it ends. With it off, nothing is published and the envelopes carry no `record`. Records live in `workflows/runs/<run id>.json`, keyed by run id, so a retried publication updates the same record instead of adding a copy.

A record carries only: the run id, workflow name, phase (`start` or `end`), driver kind, status, the end the run reached (when it reached one), the failure code of a run that failed without reaching an end, when it is a code teamai sets, such as `loop_exhausted` for a loop that ran out or `tool_unavailable` when the local driver cannot find the tool (never its message), start and end times, duration, per-step entries (step path such as `check.classify`, agent, tool, model, outcome, tokens — a mixed-agent run records each step's own fields; `tool` is kept only when it is one of the local driver's tools, and `model` is the name the driver reported, kept only when it is shaped like a model name), summary step counts (succeeded, failed, skipped, cancelled), the cumulative settled-attempt count, and the publishing username. Every entry has its `pass` number, and a step that a loop ran more than once has one entry per pass. An item of a `for_each` counts the times that item ran, so its `pass` can be lower than its step's when an earlier pass had fewer items. A cancelled step's entry has the outcome `cancelled`, and a skipped step's has none. The counts cover each step once, as the run ended. Steps of a sub-workflow that never started are not counted. Prompts, command text, input values, `env` values, outputs, diffs, secret values, local workspace paths, session ids, and hostnames have no field to land in — the record is built from an allowlist, not filtered from the run state.
A record carries only: the run id, workflow name, `sourceHash`, optional `teamRevision`, phase (`start` or `end`), driver kind, status, the end the run reached (when it reached one), the failure code of a run that failed without reaching an end, when it is a code teamai sets, such as `loop_exhausted` for a loop that ran out or `tool_unavailable` when the local driver cannot find the tool (never its message), start and end times, duration, per-step entries (step path such as `check.classify`, agent, tool, model, outcome, tokens — a mixed-agent run records each step's own fields; `tool` is kept only when it is one of the local driver's tools, and `model` is the name the driver reported, kept only when it is shaped like a model name), summary step counts (succeeded, failed, skipped, cancelled), the cumulative settled-attempt count, and the publishing username. `sourceHash` identifies the exact workflow and prompt bytes that ran. `teamRevision` is present only when one remote-reachable team commit stores those exact bytes for the parent and every child workflow; local or unpushed edits and mixed cached revisions leave it out. Every entry has its `pass` number, and a step that a loop ran more than once has one entry per pass. An item of a `for_each` counts the times that item ran, so its `pass` can be lower than its step's when an earlier pass had fewer items. A cancelled step's entry has the outcome `cancelled`, and a skipped step's has none. The counts cover each step once, as the run ended. Steps of a sub-workflow that never started are not counted. Prompts, command text, input values, `env` values, outputs, diffs, secret values, local workspace paths, session ids, and hostnames have no field to land in — the record is built from an allowlist, not filtered from the run state.

An entry's `agent` is what that pass ran as, as observed. It holds the step's `agent:` definition and how the tool got it (`main` when the tool reads its own agent file, `prompt` when the instructions are prepended), the tool and its version, the model the launch asked for and the one the tool reported, the team commit the checkout last pulled and whether a delivered file was edited since, the project commit, the TeamAI version, and the driver kind. A value nobody observed is left out. `fingerprint` hashes the definition, tool, model, effort and team commit, so two passes with the same fingerprint ran with the same recorded parts. It is left out when the model or the team commit is unknown, or a delivered file was edited. A pass nothing observed, and a record from an older TeamAI, carry only the definition's name. While a step runs, the tool it starts, and any session that tool starts in the same checkout, skips the session-start pull when the checkout's last pull delivered completely to that tool, so the step keeps the team commit it records.

Expand Down
2 changes: 1 addition & 1 deletion docs/guide/zh-CN/workflow.md
Original file line number Diff line number Diff line change
Expand Up @@ -128,7 +128,7 @@ teamai workflow report <run> --json '{"attempt":"<attempt>","revision":1,"outcom

开启[使用统计上报](reporting.md)时,每次运行都会向团队的 `teamai-reports` 分支发布一条保护隐私的记录:开始时一次,结束时一次。关闭后不发布任何记录,返回中也不带 `record`。记录存放在 `workflows/runs/<run id>.json`,以 run id 为键,重试的发布会更新同一条记录而不是产生副本。

记录只包含:run id、工作流名、阶段(`start` 或 `end`)、驱动的 kind、状态、运行到达的 end(到达时才有)、未到达 end 就失败的运行的失败代码(仅限 teamai 设定的代码,如循环用完次数时的 `loop_exhausted`,或本地驱动找不到工具时的 `tool_unavailable`,不含失败消息)、开始和结束时间、时长、逐步条目(step 路径如 `check.classify`、agent、tool、model、outcome、tokens——混合 agent 的运行会分别记录每步自己的字段;`tool` 仅在是本地驱动的工具之一时保留,`model` 是驱动报告的名称,仅在形如模型名时保留)、步骤汇总计数(succeeded、failed、skipped、cancelled)、累计已结算 attempt 数,以及发布者用户名。每条条目都带 `pass` 编号,循环运行了多次的步骤每个 pass 各有一条条目。`for_each` 的项按该项自己运行的次数计 `pass`,所以前面某个 pass 的项较少时,项的 `pass` 可能比所属步骤的小。被取消步骤的条目 outcome 为 `cancelled`,被跳过的步骤没有 outcome。汇总计数按运行结束时的状态,每个步骤只计一次。从未开始的子工作流里的步骤不计入。提示词、命令文本、输入值、`env` 值、输出、diff、秘密值、本地工作区路径、会话 id 和主机名没有可落地的字段——记录由白名单构建,而不是从运行状态里过滤。
记录只包含:run id、工作流名、`sourceHash`、可选的 `teamRevision`、阶段(`start` 或 `end`)、驱动的 kind、状态、运行到达的 end(到达时才有)、未到达 end 就失败的运行的失败代码(仅限 teamai 设定的代码,如循环用完次数时的 `loop_exhausted`,或本地驱动找不到工具时的 `tool_unavailable`,不含失败消息)、开始和结束时间、时长、逐步条目(step 路径如 `check.classify`、agent、tool、model、outcome、tokens——混合 agent 的运行会分别记录每步自己的字段;`tool` 仅在是本地驱动的工具之一时保留,`model` 是驱动报告的名称,仅在形如模型名时保留)、步骤汇总计数(succeeded、failed、skipped、cancelled)、累计已结算 attempt 数,以及发布者用户名。`sourceHash` 标识实际运行的工作流和提示文件的精确字节。只有一个远端可达的团队提交同时保存父工作流和所有子工作流的这些精确字节时,才写入 `teamRevision`;本地或未推送的修改以及混合缓存版本都会省略它。每条条目都带 `pass` 编号,循环运行了多次的步骤每个 pass 各有一条条目。`for_each` 的项按该项自己运行的次数计 `pass`,所以前面某个 pass 的项较少时,项的 `pass` 可能比所属步骤的小。被取消步骤的条目 outcome 为 `cancelled`,被跳过的步骤没有 outcome。汇总计数按运行结束时的状态,每个步骤只计一次。从未开始的子工作流里的步骤不计入。提示词、命令文本、输入值、`env` 值、输出、diff、秘密值、本地工作区路径、会话 id 和主机名没有可落地的字段——记录由白名单构建,而不是从运行状态里过滤。

条目的 `agent` 是该 pass 实际运行时观察到的组合:步骤的 `agent:` 定义及工具如何拿到它(工具自己读取 agent 文件时为 `main`,指令拼接到提示词前时为 `prompt`)、工具及其版本、启动时请求的模型和工具报告的模型、检出最近一次 pull 的团队提交以及之后是否改过已下发的文件、项目提交、TeamAI 版本和驱动的 kind。没有观察到的值不写。`fingerprint` 对定义、工具、模型、effort 和团队提交做哈希,fingerprint 相同的两个 pass 记录下来的这些部分相同。模型或团队提交未知,或改过已下发的文件时,不写 `fingerprint`。没有任何观察的 pass,以及旧版 TeamAI 写的记录,只带定义的名称。步骤运行期间,如果该检出最近一次 pull 已完整下发到步骤启动的工具,该工具以及它在同一检出中启动的会话都会跳过会话开始时的 pull,步骤因此保持它记录的团队提交。

Expand Down
Original file line number Diff line number Diff line change
@@ -1,4 +1,5 @@
import { execFileSync, spawn } from 'node:child_process';
import { createHash } from 'node:crypto';
import fs from 'node:fs/promises';
import os from 'node:os';
import path from 'node:path';
Expand Down Expand Up @@ -78,6 +79,8 @@ describe('#1083 workflow run records reach the reports branch', () => {
};

const rev = git(repo, 'rev-parse', '--short', 'HEAD').trim();
const teamRevision = git(repo, 'rev-parse', 'HEAD').trim();
const sourceHash = createHash('sha256').update(TWO_STEPS).digest('hex');
const agentFile = path.join(os.homedir(), '.claude', 'agents', 'reviewer.md');
await fs.mkdir(path.dirname(agentFile), { recursive: true });
await fs.writeFile(agentFile, '---\nname: reviewer\ndescription: Reviews builds.\nmodel: claude-opus-4-1\n---\nReview carefully.\n');
Expand Down Expand Up @@ -108,6 +111,10 @@ describe('#1083 workflow run records reach the reports branch', () => {
expect(startRecord.phase).toBe('start');
expect(startRecord.status).toBe('running');
expect(startRecord.username).toBe('alice');
expect(startRecord.sourceHash).toBe(sourceHash);
expect(startRecord.teamRevision).toBe(teamRevision);
expect(() => git(repo, 'cat-file', '-e', teamRevision)).not.toThrow();

Copy link
Copy Markdown
Collaborator

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

[P3 nit] Criterion 1 says "in a fresh clone". repo is the clone that produced the value, so cat-file -e passes there whether or not origin has the commit. Checking criteria 1 and 2 against a fresh clone of origin.git proves the claim:

Suggested change
expect(() => git(repo, 'cat-file', '-e', teamRevision)).not.toThrow();
git(sandbox, 'clone', 'origin.git', 'fresh');
const fresh = path.join(sandbox, 'fresh');
expect(() => git(fresh, 'cat-file', '-e', teamRevision)).not.toThrow();
expect(git(fresh, 'show', `${teamRevision}:workflows/build.yaml`)).toBe(TWO_STEPS);

If the real-CLI check in the PR body was also run in the working clone, a fresh clone there would match the issue's test plan.

expect(git(repo, 'show', `${teamRevision}:workflows/build.yaml`)).toBe(TWO_STEPS);

const done = await runLocalDriver(config, runId, work, {
tools: { default: 'claude' },
Expand All @@ -121,6 +128,8 @@ describe('#1083 workflow run records reach the reports branch', () => {
const endRecord = JSON.parse(endRaw) as Record<string, unknown>;
expect(endRecord.phase).toBe('end');
expect(endRecord.status).toBe('succeeded');
expect(endRecord.sourceHash).toBe(sourceHash);
expect(endRecord.teamRevision).toBe(teamRevision);
expect(endRecord.stepCounts).toEqual({ total: 3, succeeded: 3, failed: 0, skipped: 0, cancelled: 0 });
expect(endRecord.steps).toEqual([
{ step: 'one', pass: 1, outcome: 'success' },
Expand Down
Loading
Loading