feat(spec,service-automation): surface per-run flow summaries — selected / acted / skipped (#4354) (#4377)

* feat(spec,service-automation): surface per-run flow summaries — selected / acted / skipped (#4354)

`success: true` never meant "it did its job". A scheduled sweep that selects
thirty records and writes none is, from outside, identical to one with nothing
to do: same green status, same empty output, same silence, same schedule
tomorrow. Nothing anywhere separated "nothing to do" from "broken" — which is
how #4347 left three hotcrm production flows completely inert, running daily and
green for as long as they had existed.

Every terminal run now carries a `FlowRunSummary`: `selected` / `acted` /
`skipped` totals, a per-node breakdown, and which gates closed. It is surfaced
on `AutomationResult`, in `listRuns` / `getRun`, as one greppable log line, and
as queryable `selected_count` / `acted_count` / `skipped_count` columns on
`sys_automation_run` — so `selected > 0 AND acted = 0` over consecutive runs is
something an operator can alert on rather than notice.

The counts are declared by the executors (`NodeExecutionResult.metrics`), not
sniffed from node output: only the node knows whether its result is a row count,
a record, or a boolean. A closed conditional edge now records a `skipped` step
naming the gate — the event that previously left no trace at all — and that step
is explicitly not a run, so the ADR-0044 re-entry guard and the per-node `runs`
counts are unchanged.

Two details decide whether the numbers can be trusted: the summary is folded
from the full step log before history compaction (and rehydrated from
`summary_json`, never re-folded from the compacted steps), and `subflow` / `map`
children roll up into their parent — synchronously via `metrics`, and via
`creditChildRun` when the child paused and bubbles back.

Additive throughout: `summary` is optional everywhere, legacy rows keep `null`
counts rather than `0`, and no execution behaviour changes. The log line
defaults to `info`, with `runSummaryLog: 'debug' | 'off'` to lower the volume
without turning the measurement off.

Verified: service-automation 525/46 (26 new), spec 7165/279 (5 new), runtime
974/68, plugin-approvals 330/13; all 8 spec `check:generated` gates plus
`check:liveness` and `check:exported-any`; eslint clean; `tsc --noEmit` at the
ledgered 2 pre-existing errors.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SzJqHp1yXjri1tqKP8WoL4

* fix(service-automation,docs): raw NUL in the gate key, and the strictness-ledger count (#4354)

Two CI gates on the previous commit, both real:

- `check:nul-bytes` — the gate-dedup key joined its parts with LITERAL NUL bytes
  instead of the escape sequence. Byte-identical at runtime, but a raw NUL makes
  ripgrep treat the whole file as binary and silently return zero matches, so
  `run-summary.ts` would have dropped out of code search and every grep-based
  lint. Written as the escape now, with the reason the joiner is NUL at all
  spelled out (an edge id is author-supplied text, so a printable separator
  could collide two distinct gates).
- `check:strictness-ledger` — `automation/execution.zod.ts` gained 5 `z.object(`
  sites (the run-summary family), so its ledger row and the section total were
  stale. Re-read: they are engine-emitted telemetry that nobody authors, so the
  row's `wire` verdict covers them unchanged; count updated to 13 (98 for the
  section) with the additions named.

Verified: service-automation 525/46; `pnpm lint`, `check:nul-bytes` and
`check:slot-lookup` clean; all seven pure spec audits (liveness, empty-state,
variant-docs, strictness-ledger, react-conformance, skill-examples,
exported-any) green.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01SzJqHp1yXjri1tqKP8WoL4

---------

Co-authored-by: Claude <noreply@anthropic.com>

Author: os-zhuang

.changeset/flow-run-summaries.md

@@ -0,0 +1,99 @@
+---
+"@objectstack/spec": minor
+"@objectstack/service-automation": minor
+---
+
+feat(spec,service-automation): every flow run reports what it actually did — selected / acted / skipped (#4354)
+
+`success: true` never meant "it did its job". A scheduled sweep that selects
+thirty records and writes none is, from outside, **identical** to one with
+nothing to do: same green status, same empty output, same silence, same schedule
+tomorrow. There was no signal anywhere that separated "nothing to do" from
+"broken".
+
+That is not theoretical. #4347 left three hotcrm production flows completely
+inert — the stalled-deal sweep found every stalled deal and nudged nobody, the
+renewal sweep booked nothing, the campaign action enrolled no leads. They ran
+daily, on time, green, for as long as they had existed, and were caught only by
+adding tests that assert on records written. Automation is exactly the category
+where nobody is watching: a UI bug files a ticket within the hour, a dead sweep
+files nothing, and the longer it runs the more normal the silence looks.
+
+**Every terminal run now carries a `FlowRunSummary`** — on the
+`AutomationResult`, on the run in `listRuns` / `getRun`, in the log, and in the
+database:
+
+```
+[automation] run flow=stalled_deal_sweep run=run_a1b2 status=completed durationMs=142 selected=30 acted=0 skipped=30 gate=check_stalled->send_nudge:30
+```
+
+- `selected` — records read by the run's data nodes
+- `acted` — records created / updated / deleted, plus effects dispatched
+  (notifications delivered)
+- `skipped` — node executions a closed gate prevented, one per loop iteration
+  whose conditional edge evaluated false
+- `nodes[]` — per-node terminal status with `runs` / `failures` / `skipped`
+- `gates[]` — which gates closed and how often, most-skipped first
+
+**The counts are declared, not sniffed.** Executors report
+`NodeExecutionResult.metrics`, because only the node knows what its result
+*means*: `update_record`'s is a row count on a bulk write and a record on a by-id
+one, `delete_record`'s can be a boolean, `notify`'s is a delivery count. An
+engine inferring from output shapes would be guessing, and a machine-readable
+count that guesses is worse than none. A node that touches no records
+(`decision`, `assignment`) reports nothing — absent is not `0`.
+
+**The gate is named.** A conditional out-edge that evaluates false now records a
+`skipped` step tagged with the gate that closed. That event previously left no
+trace at all, which is why #4347 was invisible: the flow selected every row and
+the loop-body edge never opened. A skipped step is explicitly *not* a run — the
+ADR-0044 re-entry guard, per-node `runs`, and node status all exclude it, so a
+new observability signal cannot change execution semantics.
+
+**Queryable, so it can be alerted on rather than noticed.**
+`sys_automation_run` gains `selected_count` / `acted_count` / `skipped_count`
+columns plus a `summary_json` breakdown:
+
+```typescript
+const suspect = await engine.find('sys_automation_run', {
+  where: { status: 'completed', selected_count: { $gt: 0 }, acted_count: 0 },
+  orderBy: [{ field: 'started_at', order: 'desc' }],
+});
+```
+
+`selected > 0 && acted == 0` over consecutive runs is a near-perfect
+broken-sweep detector. Columns, not JSON: an operator can only alert on what is
+filterable. Rows written before this carry `null`, never `0` — "not measured"
+must not read as "measured zero", or every legacy row is a false alarm the first
+time someone writes that query.
+
+Two details that decide whether the numbers can be trusted. The summary is
+folded from the **full** step log before history compaction, so a
+5000-iteration sweep does not silently report the ~200 steps that fit in
+`steps_json`; and rehydration reads the persisted `summary_json` rather than
+re-folding those compacted steps. A `subflow` rolls its child's totals into its
+parent, so a sweep that delegates its writes is not read as inert — the child
+keeps its own run row, and the parent's summary answers "what did this run
+cause".
+
+Additive throughout: `summary` is optional everywhere it appears, existing runs
+and stores keep working, and no execution behaviour changes. The one-line log
+defaults to `info` — a line nobody sees at their production level is the same
+non-signal this closes — with `AutomationServicePlugin`'s
+`runSummaryLog: 'debug' | 'off'` to turn the volume down on a very
+high-frequency flow without turning the measurement off.
+
+New spec exports: `FlowRunSummarySchema`, `FlowRunNodeSummarySchema`,
+`FlowRunGateSummarySchema`, `ExecutionStepMetricsSchema`,
+`ExecutionStepSkipReasonSchema` (+ inferred types); `ExecutionLog.summary` and
+`ExecutionStepLog.metrics` / `.skippedBy`. `service-automation` exports
+`summarizeRun` / `formatRunSummaryLine` so a host building its own surface
+reuses the platform's definition instead of re-deriving one.
+
+Does not fix #4347 itself — this is the instrument that would have caught it.
+
+Verified: `@objectstack/service-automation` **522 tests / 46 files** (23 new),
+`@objectstack/spec` **7165 / 279** (5 new), `@objectstack/runtime` **974 / 68**,
+`@objectstack/plugin-approvals` **330 / 13**; all eight `@objectstack/spec`
+`check:generated` gates plus `check:liveness` and `check:exported-any`; and
+`tsc --noEmit` on service-automation at its ledgered 2 pre-existing errors.

content/docs/automation/flows.mdx

@@ -579,6 +579,61 @@ panel. Recent runs are held in an in-memory ring buffer; terminal runs
 history with a bounded step log, so `listRuns` / `getRun` still report a run's
 status, steps, and failure reason after a restart or ring-buffer eviction.
 
+### Run summaries
+
+A run that reports `success: true` has not told you it did its job. A scheduled
+sweep that selects thirty records and writes none looks *identical* to one with
+nothing to do: same green status, same empty output, same silence. Every
+terminal run therefore carries a **summary** — on the `AutomationResult`, on the
+run in `listRuns` / `getRun`, and in the log:
+
+```
+[automation] run flow=stalled_deal_sweep run=run_a1b2 status=completed durationMs=142 selected=30 acted=0 skipped=30 gate=check_stalled->send_nudge:30
+```
+
+| Field | Meaning |
+| :--- | :--- |
+| `selected` | Records **read** by the run's data nodes |
+| `acted` | Records **created / updated / deleted**, plus effects dispatched (notifications delivered) |
+| `skipped` | Node executions a **closed gate** prevented — one per loop iteration whose conditional edge evaluated false |
+| `nodes[]` | Per-node terminal status with `runs` / `failures` / `skipped` and its own selected/acted |
+| `gates[]` | Which gates closed and how often, most-skipped first |
+
+The counts come from the node executors themselves — `get_record` reports what
+it matched, the write nodes report what the data engine actually changed — not
+from the engine guessing at a node's output shape. A node that touches no
+records (`decision`, `assignment`) reports nothing at all, which is different
+from reporting zero.
+
+The same counts land on `sys_automation_run` as **queryable columns**
+(`selected_count`, `acted_count`, `skipped_count`, plus a `summary_json`
+breakdown), so a broken sweep is something you can alert on rather than notice:
+
+```typescript
+// Runs that selected work and did none of it, newest first.
+const suspect = await engine.find('sys_automation_run', {
+  where: {
+    status: 'completed',
+    selected_count: { $gt: 0 },
+    acted_count: 0,
+    started_at: { $gte: since },
+  },
+  orderBy: [{ field: 'started_at', order: 'desc' }],
+});
+```
+
+`selected > 0 && acted == 0` over several consecutive runs is a near-perfect
+broken-sweep detector — the case that is otherwise invisible, because nobody is
+watching automation until it has already been dead for a month. A single such
+run is not proof of anything: a sweep whose work is all already done reports the
+same thing legitimately, which is why the signal is *consecutive* runs, and why
+the platform reports the counts rather than raising the alarm itself.
+
+Rows written before summaries existed carry `null` counts, not `0` — "not
+measured" must not read as "measured zero". The log line defaults to `info`;
+`AutomationServicePlugin`'s `runSummaryLog: 'debug' | 'off'` turns the volume
+down on a very high-frequency flow without turning the measurement off.
+
 ## Edges
 
 Edges connect nodes and define the execution path:

content/docs/references/automation/execution.mdx

@@ -22,8 +22,8 @@ AWS Step Functions execution logs.
 ## TypeScript Usage
 
 ```typescript
-import { Checkpoint, ConcurrencyPolicy, ExecutionError, ExecutionErrorSeverity, ExecutionLog, ExecutionStatus, ExecutionStepLog, ScheduleState } from '@objectstack/spec/automation';
-import type { Checkpoint, ConcurrencyPolicy, ExecutionError, ExecutionErrorSeverity, ExecutionLog, ExecutionStatus, ExecutionStepLog, ScheduleState } from '@objectstack/spec/automation';
+import { Checkpoint, ConcurrencyPolicy, ExecutionError, ExecutionErrorSeverity, ExecutionLog, ExecutionStatus, ExecutionStepLog, ExecutionStepMetrics, ExecutionStepSkipReason, FlowRunGateSummary, FlowRunNodeSummary, FlowRunSummary, ScheduleState } from '@objectstack/spec/automation';
+import type { Checkpoint, ConcurrencyPolicy, ExecutionError, ExecutionErrorSeverity, ExecutionLog, ExecutionStatus, ExecutionStepLog, ExecutionStepMetrics, ExecutionStepSkipReason, FlowRunGateSummary, FlowRunNodeSummary, FlowRunSummary, ScheduleState } from '@objectstack/spec/automation';
 
 // Validate data
 const result = Checkpoint.parse(data);
@@ -108,6 +108,7 @@ const result = Checkpoint.parse(data);
 | **status** | `Enum<'pending' \| 'running' \| 'paused' \| 'completed' \| 'failed' \| 'cancelled' \| 'timed_out' \| 'retrying'>` | ✅ | Current execution status |
 | **trigger** | `{ type: string; recordId?: string; object?: string; userId?: string; … }` | ✅ | What triggered this execution |
 | **steps** | `{ nodeId: string; nodeType: string; nodeLabel?: string; status: Enum<'success' \| 'failure' \| 'skipped'>; … }[]` | ✅ | Ordered list of executed steps |
+| **summary** | `{ selected: integer; acted: integer; skipped: integer; nodes: { nodeId: string; nodeType: string; nodeLabel?: string; status: Enum<'success' \| 'failure' \| 'skipped'>; … }[]; … }` | optional | Per-run rollup: records selected / acted on, gate skips, per-node status |
 | **variables** | `Record<string, any>` | optional | Final state of flow variables |
 | **startedAt** | `string` | ✅ | Execution start timestamp |
 | **completedAt** | `string` | optional | Execution completion timestamp |
@@ -154,6 +155,83 @@ const result = Checkpoint.parse(data);
 | **parentNodeId** | `string` | optional | Enclosing structured-region container node ID (loop/parallel/try_catch) |
 | **iteration** | `integer` | optional | Zero-based loop iteration or parallel branch index of the enclosing region |
 | **regionKind** | `string` | optional | Region kind the step ran in: loop-body \| parallel-branch \| try \| catch |
+| **metrics** | `{ selected?: integer; acted?: integer }` | optional | Records this step selected / acted on, as reported by the node executor |
+| **skippedBy** | `{ nodeId: string; edgeId?: string; label?: string }` | optional | The gate that closed, when `status` is `skipped` |
+
+
+---
+
+## ExecutionStepMetrics
+
+### Properties
+
+| Property | Type | Required | Description |
+| :--- | :--- | :--- | :--- |
+| **selected** | `integer` | optional | Records this node READ or matched (a `get_record` query, a lookup) |
+| **acted** | `integer` | optional | Records this node WROTE (created / updated / deleted) or effects it dispatched (notifications delivered) |
+
+
+---
+
+## ExecutionStepSkipReason
+
+### Properties
+
+| Property | Type | Required | Description |
+| :--- | :--- | :--- | :--- |
+| **nodeId** | `string` | ✅ | Node whose out-edge did not open (the gate) |
+| **edgeId** | `string` | optional | Edge whose condition evaluated false |
+| **label** | `string` | optional | Edge label, when the flow names its branches |
+
+
+---
+
+## FlowRunGateSummary
+
+### Properties
+
+| Property | Type | Required | Description |
+| :--- | :--- | :--- | :--- |
+| **nodeId** | `string` | ✅ | Node whose out-edge did not open (the gate) |
+| **targetNodeId** | `string` | ✅ | Node the closed edge would have run |
+| **edgeId** | `string` | optional | Edge whose condition evaluated false |
+| **label** | `string` | optional | Edge label, when the flow names its branches |
+| **skipped** | `integer` | ✅ | Times this gate evaluated false (once per loop iteration) |
+
+
+---
+
+## FlowRunNodeSummary
+
+### Properties
+
+| Property | Type | Required | Description |
+| :--- | :--- | :--- | :--- |
+| **nodeId** | `string` | ✅ | Node ID |
+| **nodeType** | `string` | ✅ | Node action type (e.g., "get_record", "decision") |
+| **nodeLabel** | `string` | optional | Human-readable node label |
+| **status** | `Enum<'success' \| 'failure' \| 'skipped'>` | ✅ | Terminal status of the node across the run — `failure` if any execution failed, else `success` if any succeeded, else `skipped` |
+| **runs** | `integer` | ✅ | Times the node executed (loop iterations and parallel branches each count) |
+| **failures** | `integer` | ✅ | Executions that failed |
+| **skipped** | `integer` | ✅ | Times a closed gate kept this node from running at all |
+| **selected** | `integer` | optional | Records read across every execution — omitted for a node that reads none |
+| **acted** | `integer` | optional | Records written / effects dispatched across every execution — omitted for a node that writes none |
+
+
+---
+
+## FlowRunSummary
+
+### Properties
+
+| Property | Type | Required | Description |
+| :--- | :--- | :--- | :--- |
+| **selected** | `integer` | ✅ | Total records read by the run |
+| **acted** | `integer` | ✅ | Total records written / effects dispatched by the run |
+| **skipped** | `integer` | ✅ | Total node executions a closed gate prevented |
+| **nodes** | `{ nodeId: string; nodeType: string; nodeLabel?: string; status: Enum<'success' \| 'failure' \| 'skipped'>; … }[]` | ✅ | Per-node breakdown, in first-execution order |
+| **gates** | `{ nodeId: string; targetNodeId: string; edgeId?: string; label?: string; … }[]` | ✅ | Gates that closed during the run, most-skipped first |
+| **detailOmitted** | `boolean` | optional | Set when persistence dropped `nodes`/`gates` to keep the stored row bounded — the totals are still exact. Declared so empty arrays are never mistaken for "nothing ran". |
 
 
 ---

content/docs/releases/v17.mdx

@@ -1209,6 +1209,17 @@ is gone.
 
 ### Automation & flows
 
+- **Every terminal run reports what it did** (#4354): `selected` / `acted` /
+  `skipped` totals, a per-node breakdown, and *which gate closed* — on the run
+  result, in `listRuns` / `getRun`, as one greppable log line
+  (`selected=30 acted=0 skipped=30 gate=check_stalled->send_nudge:30`), and as
+  queryable `selected_count` / `acted_count` / `skipped_count` columns on
+  `sys_automation_run`. `success: true` never meant "it did its job": a sweep
+  that selected thirty records and wrote none was indistinguishable from one
+  with nothing to do, which is how three inert production flows ran green for
+  as long as they had existed. `selected > 0 && acted == 0` over consecutive
+  runs is now the detector. See
+  [Run summaries](/docs/automation/flows#run-summaries).
 - **`record-after-write`** fires one flow on create **or** update (#3427), with
   `previous` bound as `null` on the create leg so start conditions can
   discriminate.

docs/audits/2026-07-unknown-key-strictness-ledger.md

@@ -165,14 +165,14 @@ tightening (the #4001 "sharing-rule lesson": candidates, not verdicts).
 | `external-catalog.zod.ts` | 4 | wire (p) | |
 | `field-value.zod.ts` / `seed.zod.ts` / `validation.zod.ts` | 1 ea | mixed (p) | |
 
-### `automation/` — 93 sites
+### `automation/` — 98 sites
 
 | File | Sites | Class | Note |
 |---|---|---|---|
 | `flow.zod.ts` | 11 | authorable | **strict as of #4001** (4 schemas; `FlowVersionHistorySchema` is runtime — stays tolerant) |
 | `sync.zod.ts` / `etl.zod.ts` | 12+10 | authorable (p) | authored pipelines — **candidates** |
 | `trigger-registry.zod.ts` | 11 | mixed | descriptors are code-registered (wire-ish); bindings authored |
-| `execution.zod.ts` | 8 | wire | run-state envelopes — never strict |
+| `execution.zod.ts` | 13 | wire | run-state envelopes — never strict. +5 at #4354 (the run-summary family: step metrics / skip reason / per-node / per-gate / the summary itself) — engine-emitted telemetry read by the Console and by operator queries, nobody authors them, so the `wire` verdict covers them unchanged |
 | `state-machine.zod.ts` | 7 | authorable (p) | |
 | `control-flow.zod.ts` | 6 | authorable (p) | validated structurally by `validateControlFlow` |
 | `bpmn-interop.zod.ts` | 5 | wire (p) | interop import shapes |