[feat] Publish privacy-preserving workflow run records and stats #1083

Activity

1. hsuchifeng

   added a parent issue on Oct 13, 2026

   • Discussion: Workflow Runtime Integration #818
2. hsuchifeng

   marked this as blocked on Oct 13, 2026

   • [feat] Add a minimal command-only workflow runtime #1076
3. hsuchifeng

   marked this as blocking 2 issues on Oct 13, 2026

   • [feat] Add workflow scorers and recorded quality labels #1085
   • [feat] Add a Workflow Runs page to the dashboard #1088
4. kittimzhe commented on Oct 14, 2026

   

   kittimzhe

   on Oct 14, 2026

   Contributor

   More actions

   Now that the command-only runtime base (#1096/#1097) has landed — is this one takeable by a community contributor?

   I'd like to take it, and it's a natural continuation of the stats-privacy work I did in #1056 (which touched reportUsageToTeam, the skills catalog allowlist, and the per-user stats merge on the reports branch). My implementation read, before writing any code:

   • Schema: one versioned run-record shape shared by all drivers, published exactly twice per run (start / end), with a stable run id as the logical record key.
   • Redaction: allowlist, not blocklist — the record carries workflow, step, driver, optional agent/tool/model, outcome, named end, iterations, duration, tokens, team revision, PR/CI URLs when available, and structurally cannot carry prompts, outputs, diffs, secret values, or workspace paths (nothing outside the allowlist has a field to land in).
   • Idempotency: retries upsert by run id, so "a machine that dies after start" stays visible as started-but-not-ended and a later retry cannot duplicate the logical record.
   • Queue: failed publications land in a local pending queue that survives restart and drains on the next pull (mirroring the existing reports-branch retry semantics), with record: pending surfaced on failure.
   • Stats merge: workflow totals added to per-user stats additively, preserving every unrelated field — the same preservation discipline fix(push): publish only team-shared skill names in stats #1056's fixtures pinned.

   Verification plan per the issue: focused schema/redaction/idempotency/queue/stats-merge unit tests, one real-git E2E inspecting the reports branch, bilingual reporting docs update.

   If you're reserving the series for core work, no problem — a one-word answer here settles it and I'll steer elsewhere.
5. jeff-r2026

   added

   enhancementNew feature or requestNew feature or request

   help wantedExtra attention is neededExtra attention is needed

   and removed

   help wantedExtra attention is neededExtra attention is needed

   on Oct 14, 2026
6. jeff-r2026 commented on Oct 14, 2026

   

   jeff-r2026

   on Oct 14, 2026

   Collaborator

   More actions

   Now that the command-only runtime base (#1096/#1097) has landed — is this one takeable by a community contributor?

   I'd like to take it, and it's a natural continuation of the stats-privacy work I did in #1056 (which touched reportUsageToTeam, the skills catalog allowlist, and the per-user stats merge on the reports branch). My implementation read, before writing any code:

   • Schema: one versioned run-record shape shared by all drivers, published exactly twice per run (start / end), with a stable run id as the logical record key.
   • Redaction: allowlist, not blocklist — the record carries workflow, step, driver, optional agent/tool/model, outcome, named end, iterations, duration, tokens, team revision, PR/CI URLs when available, and structurally cannot carry prompts, outputs, diffs, secret values, or workspace paths (nothing outside the allowlist has a field to land in).
   • Idempotency: retries upsert by run id, so "a machine that dies after start" stays visible as started-but-not-ended and a later retry cannot duplicate the logical record.
   • Queue: failed publications land in a local pending queue that survives restart and drains on the next pull (mirroring the existing reports-branch retry semantics), with record: pending surfaced on failure.
   • Stats merge: workflow totals added to per-user stats additively, preserving every unrelated field — the same preservation discipline fix(push): publish only team-shared skill names in stats #1056's fixtures pinned.

   Verification plan per the issue: focused schema/redaction/idempotency/queue/stats-merge unit tests, one real-git E2E inspecting the reports branch, bilingual reporting docs update.

   If you're reserving the series for core work, no problem — a one-word answer here settles it and I'll steer elsewhere.

   Please take this
7. SaulMoro commented on Oct 14, 2026

   

   SaulMoro

   on Oct 14, 2026

   Last edited by SaulMoro

   Collaborator

   More actions

   Some details that are hard to change once records are on teamai-reports.

   • Model. The record could keep tool, the concrete model, and modelAlias when the agent or step asked for one (Proposal: model aliases so one agent works in every tool #830). The concrete model comes from the report. When the tool does not report it, the delivery ledger has it, because it already records the model each agent got per tool (AgentModelRecords in src/types.ts). The alias makes implementer@strong comparable across Claude and Codex, and the concrete model lets model-pricing.ts estimate cost.
   • Run id. r plus 8 hex digits (src/workflow/store.ts) works on one machine. As a key shared by the whole team, it reaches about 1% collision odds by 10,000 runs, and a collision overwrites another member's record. A 16-digit id, with the regex still accepting the old form, avoids that.
   • Scorer labels. If stats sum the labels from [feat] Add workflow scorers and recorded quality labels #1085 per scorer, agent and model, with a source field (workflow now, session later), labels from outside workflows fit later without a migration.

   Added once #1120 was approved:

   • Where tool and model come from. tool, model, session and tokens reach the step state in feat(workflow): run prompt and agent steps with the local driver #1120, so this is easiest to build on that branch or after it merges. feat(workflow): run prompt and agent steps with the local driver #1120 already falls back to the deployed agent's model when the tool does not report one.
   • modelAlias. The deployed agent file that feat(workflow): run prompt and agent steps with the local driver #1120 reads already has the alias resolved, so the alias has to come from the team's agents/<name>.yaml. It should be captured when the run starts, the way feat(workflow): run prompt and agent steps with the local driver #1120 caches prompt_file, so a pull during the run does not change what the record says.
   • Driver and failures. DriverIdentity.workspace is an absolute local path, and a failed step's message holds the first 400 characters of stderr. The record could keep only driver.kind and failure.code.
   • Team revision. The run state keeps the workflow's sourceHash, not the team repo commit. The commit has to be read at start and stored with the run.

   Also, the token counts in reports from #1120 currently come from the agent's own result.json, so they can be invented. I raised it on #1120.

   Update: resolved in #1120 (39a7dc1). Model, session and token counts now come only from the tool's own output.
8. SaulMoro commented on Oct 14, 2026

   

   SaulMoro

   on Oct 14, 2026

   Collaborator

   More actions

   @kittimzhe the schema notes above affect the record shape, so worth a look before the first push to teamai-reports.
9. kittimzhe

   added a commit that references this issue on Oct 14, 2026

   feat(workflow): publish privacy-preserving run records to teamai-repo…

   ...

   a49d2d3
10. SaulMoro commented on Oct 14, 2026

    

    SaulMoro

    on Oct 14, 2026

    Last edited by SaulMoro

    Collaborator

    More actions

    Notes from #1078 (#1129, independent of #1126) that change what a run record reads:

    • One settlement point. A run ends only in settleRun, and report no longer settles: after a failed report, the next next skips the dependents and settles the run. Records should hook settleRun, not report.
    • Step ids are paths (ship.check.classify), so definitions are looked up by path, not by top-level id.
    • New statuses. Steps can be skipped, and a run can carry end (the named end it reached). Counts that only know succeeded and failed undercount.
    • New run fields. A run stores inputs and a snapshot of every child definition. Neither belongs in a published record without a privacy decision.
    • One timestamp. settleRun and the save in withRun each stamp updatedAt. Pass one now through, so a record and the run file agree.
    • A failed workflow step has no failure or message of its own; the child steps carry theirs.

    The adaptation is ready on https://github.com/SaulMoro/teamai-cli/commits/records-on-graph-runs (details in #1126 (comment)); whichever of #1126 and #1129 merges second takes it.

    The engine rules that span files are in docs/dev/workflow-engine.md, added by #1129.
11. kittimzhe

    added a commit that references this issue on Oct 15, 2026

    feat(workflow): publish privacy-preserving run records to teamai-repo…

    ...

    d36d28e
12. jeff-r2026

    added a commit that references this issue on Oct 15, 2026

    feat(workflow): publish privacy-preserving run records to teamai-repo…

    ...

    d25b7ae
13. SaulMoro

    marked this as blocking 2 issues on Oct 15, 2026

    • [feat] Define the team revision once for run records, labels and snapshots #1163
    • [feat] Deliver workflow branches as pull requests #1090
14. jeff-r2026

    closed this as completedin #1161on Oct 16, 2026