feat(spec,service-automation): a run says when its acted count is incomplete, instead of guessing (#4354)

.changeset/run-summary-uncountable-effects.md

@@ -0,0 +1,72 @@
+---
+"@objectstack/spec": minor
+"@objectstack/service-automation": minor
+---
+
+feat(spec,service-automation): a run says when its `acted` count is incomplete, instead of guessing (#4354)
+
+#4354 shipped `selected` / `acted` counts on every flow run, sourced from the
+executors that know what they did. Four node types were left out — and the gap
+was not cosmetic: `connector_action`, `http` and `script` are how a flow acts on
+anything *outside* the platform, so a sweep whose whole job runs through them
+reported `acted: 0` and looked exactly like the dead sweep the counter exists to
+find. A detector that fires on healthy runs is worse than no detector: operators
+tune it out, and then it is not watching the flows that really did stop.
+
+Closing it needed a third answer, because for two of those nodes the platform
+genuinely cannot know:
+
+**`connector_action` — unknowable, and now it says so.**
+`ConnectorActionDescriptor` declares `key` / `label` / `description` /
+`inputSchema` / `outputSchema` and *nothing* about whether the action reads or
+writes, so `crm.push_opportunity` and `crm.lookup_account` are the same shape to
+the runtime. `acted: 0` understates the create; `acted: 1` overstates the
+lookup and makes the alert never fire — #4354's original bug, one layer out.
+The executor reports `metrics: { unmeasuredEffect: true }` instead, and the run
+carries an `unmeasured` tally. Filed #4395 to let a connector declare its effect
+kind, which would turn this into a real count.
+
+**`http` — knowable, and now counted.** The method says it:
+`GET`/`HEAD`/`OPTIONS` report a real `acted: 0` (a read cannot write); a mutating
+call the upstream accepted reports `acted: 1`; `durable: true` reports `acted: 1`
+because the outbox row is a durable effect this run caused. A mutating call that
+was *rejected or timed out* reports `unmeasured` — a 500 can arrive after the
+write landed, and claiming zero there would let a run swear it changed nothing
+when it had.
+
+**`script` — deliberately unchanged.** A registered function is contractually
+pure ("Data I/O stays on the flow graph — the function itself does no writes"),
+so every write it causes is a downstream node counting itself and "reports no
+record metrics" is accurate rather than a guess. Nothing *enforces* that purity,
+so a function that writes behind the platform's back under-reports its run —
+filed as #4396 rather than papered over here, because a blanket
+`unmeasuredEffect` on `script` would suppress the signal on every flow that
+calls any function in order to accommodate one contract violation.
+
+**The alert gains a clause.** `selected > 0 AND acted = 0` becomes
+`selected > 0 AND acted = 0 AND unmeasured = 0`, and `sys_automation_run` gains
+an `unmeasured_count` column to serve it. Without that third clause the alert
+fires on every healthy connector-driven flow. The log line gains
+`unmeasured=N` — only when non-zero, since its *presence* is what a reader must
+not miss: `acted=0` on a line that also says `unmeasured=3` means "cannot tell",
+not "did nothing".
+
+`unmeasured` propagates through `subflow` and `map` roll-ups (and through
+`creditChildRun` for a child that paused), so a parent whose child dispatched an
+uncountable effect knows its own `acted` is incomplete. N uncountable effects in
+a child collapse to one flag on the parent's step — the child keeps the real
+count in its own run row, and the question this feeds is boolean.
+
+`FlowRunSummary.unmeasured` is optional and `undefined` is **not** `0`: a run
+recorded before this existed did not track uncountable effects at all, and
+defaulting it to zero would tell an operator "fully measured" about a run nobody
+measured. Same rule the `null` count columns already follow.
+
+Additive: new optional fields only, no new exports, no execution behaviour
+changes.
+
+Verified: `@objectstack/service-automation` **546 tests / 47 files** (21 new),
+`@objectstack/spec` **7193 / 281** (2 new); all 8 `check:generated` gates plus
+the seven pure audits (liveness, empty-state, variant-docs, strictness-ledger,
+react-conformance, skill-examples, exported-any); `check:nul-bytes` and eslint
+clean.

content/docs/automation/flows.mdx

@@ -596,6 +596,7 @@ run in `listRuns` / `getRun`, and in the log:
 | `selected` | Records **read** by the run's data nodes |
 | `acted` | Records **created / updated / deleted**, plus effects dispatched (notifications delivered) |
 | `skipped` | Node executions a **closed gate** prevented — one per loop iteration whose conditional edge evaluated false |
+| `unmeasured` | Executions that reached something the platform **cannot count** — see below |
 | `nodes[]` | Per-node terminal status with `runs` / `failures` / `skipped` and its own selected/acted |
 | `gates[]` | Which gates closed and how often, most-skipped first |
 
@@ -605,9 +606,32 @@ from the engine guessing at a node's output shape. A node that touches no
 records (`decision`, `assignment`) reports nothing at all, which is different
 from reporting zero.
 
+#### When the platform cannot count
+
+Some nodes reach outside the platform, and for those `acted` has a third
+possible answer. A `connector_action` dispatches to an external system through a
+descriptor that declares nothing about whether the action reads or writes, so
+counting it `0` would understate a Salesforce create and counting it `1` would
+overstate a lookup — and the overstatement is the worse one, because it makes
+the broken-sweep alert never fire. Those executions increment `unmeasured`
+instead:
+
+| Node | Reported |
+| :--- | :--- |
+| `http`, `GET`/`HEAD`/`OPTIONS` | `acted: 0` — a read can never write |
+| `http`, mutating method, response OK | `acted: 1` |
+| `http`, mutating method, rejected / timed out | `unmeasured` — a 500 can arrive after the write landed |
+| `http`, `durable: true` | `acted: 1` — the outbox row is a real, durable effect |
+| `connector_action` | `unmeasured` |
+| `script` | nothing — a registered function is **contractually pure**: data I/O stays on the flow graph, so every write it causes is a downstream node that counts itself |
+
+`unmeasured` propagates through `subflow` and `map` roll-ups, so a parent whose
+child dispatched an uncountable effect knows its own `acted` is incomplete.
+
 The same counts land on `sys_automation_run` as **queryable columns**
-(`selected_count`, `acted_count`, `skipped_count`, plus a `summary_json`
-breakdown), so a broken sweep is something you can alert on rather than notice:
+(`selected_count`, `acted_count`, `skipped_count`, `unmeasured_count`, plus a
+`summary_json` breakdown), so a broken sweep is something you can alert on
+rather than notice:
 
 ```typescript
 // Runs that selected work and did none of it, newest first.
@@ -616,18 +640,22 @@ const suspect = await engine.find('sys_automation_run', {
     status: 'completed',
     selected_count: { $gt: 0 },
     acted_count: 0,
+    // Without this clause the alert fires on every healthy connector-driven
+    // flow: those runs report acted 0 because the count is INCOMPLETE, not zero.
+    unmeasured_count: 0,
     started_at: { $gte: since },
   },
   orderBy: [{ field: 'started_at', order: 'desc' }],
 });
 ```
 
-`selected > 0 && acted == 0` over several consecutive runs is a near-perfect
-broken-sweep detector — the case that is otherwise invisible, because nobody is
-watching automation until it has already been dead for a month. A single such
-run is not proof of anything: a sweep whose work is all already done reports the
-same thing legitimately, which is why the signal is *consecutive* runs, and why
-the platform reports the counts rather than raising the alarm itself.
+`selected > 0 && acted == 0 && unmeasured == 0` over several consecutive runs is
+a near-perfect broken-sweep detector — the case that is otherwise invisible,
+because nobody is watching automation until it has already been dead for a
+month. A single such run is not proof of anything: a sweep whose work is all
+already done reports the same thing legitimately, which is why the signal is
+*consecutive* runs, and why the platform reports the counts rather than raising
+the alarm itself.
 
 Rows written before summaries existed carry `null` counts, not `0` — "not
 measured" must not read as "measured zero". The log line defaults to `info`;

content/docs/references/automation/execution.mdx

@@ -108,7 +108,7 @@ const result = Checkpoint.parse(data);
 | **status** | `Enum<'pending' \| 'running' \| 'paused' \| 'completed' \| 'failed' \| 'cancelled' \| 'timed_out' \| 'retrying'>` | ✅ | Current execution status |
 | **trigger** | `{ type: string; recordId?: string; object?: string; userId?: string; … }` | ✅ | What triggered this execution |
 | **steps** | `{ nodeId: string; nodeType: string; nodeLabel?: string; status: Enum<'success' \| 'failure' \| 'skipped'>; … }[]` | ✅ | Ordered list of executed steps |
-| **summary** | `{ selected: integer; acted: integer; skipped: integer; nodes: { nodeId: string; nodeType: string; nodeLabel?: string; status: Enum<'success' \| 'failure' \| 'skipped'>; … }[]; … }` | optional | Per-run rollup: records selected / acted on, gate skips, per-node status |
+| **summary** | `{ selected: integer; acted: integer; skipped: integer; unmeasured?: integer; … }` | optional | Per-run rollup: records selected / acted on, gate skips, per-node status |
 | **variables** | `Record<string, any>` | optional | Final state of flow variables |
 | **startedAt** | `string` | ✅ | Execution start timestamp |
 | **completedAt** | `string` | optional | Execution completion timestamp |
@@ -155,7 +155,7 @@ const result = Checkpoint.parse(data);
 | **parentNodeId** | `string` | optional | Enclosing structured-region container node ID (loop/parallel/try_catch) |
 | **iteration** | `integer` | optional | Zero-based loop iteration or parallel branch index of the enclosing region |
 | **regionKind** | `string` | optional | Region kind the step ran in: loop-body \| parallel-branch \| try \| catch |
-| **metrics** | `{ selected?: integer; acted?: integer }` | optional | Records this step selected / acted on, as reported by the node executor |
+| **metrics** | `{ selected?: integer; acted?: integer; unmeasuredEffect?: boolean }` | optional | Records this step selected / acted on, as reported by the node executor |
 | **skippedBy** | `{ nodeId: string; edgeId?: string; label?: string }` | optional | The gate that closed, when `status` is `skipped` |
 
 
@@ -169,6 +169,7 @@ const result = Checkpoint.parse(data);
 | :--- | :--- | :--- | :--- |
 | **selected** | `integer` | optional | Records this node READ or matched (a `get_record` query, a lookup) |
 | **acted** | `integer` | optional | Records this node WROTE (created / updated / deleted) or effects it dispatched (notifications delivered) |
+| **unmeasuredEffect** | `boolean` | optional | This execution may have caused an effect the platform cannot count (an external write through a connector). NOT interchangeable with `acted: 0` — it says the count is unknown, not that it is zero. |
 
 
 ---
@@ -216,6 +217,7 @@ const result = Checkpoint.parse(data);
 | **skipped** | `integer` | ✅ | Times a closed gate kept this node from running at all |
 | **selected** | `integer` | optional | Records read across every execution — omitted for a node that reads none |
 | **acted** | `integer` | optional | Records written / effects dispatched across every execution — omitted for a node that writes none |
+| **unmeasured** | `integer` | optional | Executions that may have caused an effect the platform cannot count (see ExecutionStepMetrics.unmeasuredEffect) |
 
 
 ---
@@ -229,6 +231,7 @@ const result = Checkpoint.parse(data);
 | **selected** | `integer` | ✅ | Total records read by the run |
 | **acted** | `integer` | ✅ | Total records written / effects dispatched by the run |
 | **skipped** | `integer` | ✅ | Total node executions a closed gate prevented |
+| **unmeasured** | `integer` | optional | Total executions that may have caused an effect the platform cannot count. Absent = not tracked (an older run), which is not the same as zero. |
 | **nodes** | `{ nodeId: string; nodeType: string; nodeLabel?: string; status: Enum<'success' \| 'failure' \| 'skipped'>; … }[]` | ✅ | Per-node breakdown, in first-execution order |
 | **gates** | `{ nodeId: string; targetNodeId: string; edgeId?: string; label?: string; … }[]` | ✅ | Gates that closed during the run, most-skipped first |
 | **detailOmitted** | `boolean` | optional | Set when persistence dropped `nodes`/`gates` to keep the stored row bounded — the totals are still exact. Declared so empty arrays are never mistaken for "nothing ran". |

packages/services/service-automation/src/builtin/connector-nodes.ts

@@ -88,11 +88,23 @@ export function registerConnectorNodes(engine: AutomationEngine, ctx: PluginCont
 
             try {
                 const output = await handler((cfg.input ?? {}) as Record<string, unknown>, handlerCtx);
-                return { success: true, output };
+                // #4354 — the action reached an external system and the platform
+                // cannot say what it did there: `ConnectorActionDescriptor`
+                // declares `key` / `label` / `description` / `inputSchema` /
+                // `outputSchema` and NOTHING about whether the action reads or
+                // writes. `acted: 0` would understate a Salesforce create;
+                // `acted: 1` would overstate a lookup and make the broken-sweep
+                // alert never fire — the original bug back again. Report the
+                // honest third answer: this run's `acted` is incomplete.
+                // #4395 proposes declaring the effect kind on the descriptor,
+                // which would turn this into a real count.
+                return { success: true, output, metrics: { unmeasuredEffect: true } };
             } catch (err) {
                 return {
                     success: false,
                     error: `connector_action(${cfg.connectorId}.${cfg.actionId}) failed: ${(err as Error).message}`,
+                    // A handler that threw may still have reached the upstream.
+                    metrics: { unmeasuredEffect: true },
                 };
             }
         },

packages/services/service-automation/src/builtin/http-nodes.ts

@@ -131,7 +131,9 @@ export function registerHttpNodes(engine: AutomationEngine, ctx: PluginContext):
                             timeoutMs,
                             payload: body ?? {},
                         });
-                        return { success: true, output: { deliveryId, enqueued: true } };
+                        // #4354 — the outbox row IS a durable effect this run
+                        // caused, even though the upstream call happens later.
+                        return { success: true, output: { deliveryId, enqueued: true }, metrics: { acted: 1 } };
                     } catch (err) {
                         return { success: false, error: `http (durable) failed to enqueue: ${(err as Error).message}` };
                     }
@@ -144,6 +146,10 @@ export function registerHttpNodes(engine: AutomationEngine, ctx: PluginContext):
 
             // ── Request/response mode (default; preserves http_request) ───────
             const method = cfg.method ?? 'GET';
+            // #4354 — unlike a connector action, an HTTP call's effect IS
+            // knowable: the method says it. A GET reads and can never write, so
+            // it reports a real `0`; anything else is a mutating call.
+            const reads = /^(GET|HEAD|OPTIONS)$/i.test(method);
             const controller = new AbortController();
             const timer = timeoutMs ? setTimeout(() => controller.abort(), timeoutMs) : undefined;
             try {
@@ -158,11 +164,26 @@ export function registerHttpNodes(engine: AutomationEngine, ctx: PluginContext):
                     success: response.ok,
                     output: { response: data, status: response.status },
                     error: response.ok ? undefined : `HTTP ${response.status}`,
+                    // A mutating call the upstream ACCEPTED is one effect. One it
+                    // rejected is unknown, not zero: a 500 can arrive after the
+                    // write landed, and claiming `0` there would let a run report
+                    // it changed nothing while it had.
+                    metrics: reads
+                        ? { acted: 0 }
+                        : response.ok
+                            ? { acted: 1 }
+                            : { unmeasuredEffect: true },
                 };
             } catch (err) {
                 const e = err as { name?: string; message?: string };
                 const msg = e?.name === 'AbortError' ? `timeout after ${timeoutMs}ms` : e?.message ?? String(err);
-                return { success: false, error: `http: ${msg}` };
+                // A timed-out or aborted mutating request may well have landed —
+                // the response is what we lost, not necessarily the write.
+                return {
+                    success: false,
+                    error: `http: ${msg}`,
+                    metrics: reads ? { acted: 0 } : { unmeasuredEffect: true },
+                };
             } finally {
                 if (timer) clearTimeout(timer);
             }

packages/services/service-automation/src/builtin/map-node.ts

@@ -138,6 +138,7 @@ export function registerMapNode(engine: AutomationEngine, ctx: PluginContext): v
       // child run bubbles back (AutomationEngine.creditChildRun).
       let selected = 0;
       let acted = 0;
+      let unmeasured = false;
 
       // Drive items in order. Synchronous items advance inline; a pausing item
       // suspends the run and is resumed via re-entry.
@@ -175,22 +176,32 @@ export function registerMapNode(engine: AutomationEngine, ctx: PluginContext): v
           // Mark this item started and suspend; the engine re-enters on bubble.
           state.started = idx + 1;
           variables.set(stateKey, state);
-          return { success: true, suspend: true, correlation: `map:${child.runId}`, metrics: { selected, acted } };
+          return {
+            success: true, suspend: true, correlation: `map:${child.runId}`,
+            metrics: { selected, acted, ...(unmeasured ? { unmeasuredEffect: true } : {}) },
+          };
         }
         if (!child.success) {
           return {
             success: false,
             error: `map '${node.id}': item ${idx} (subflow '${flowName}') failed: ${child.error ?? 'unknown error'}`,
             // Items that already succeeded wrote real rows; a later item's
             // failure must not erase them from the run's totals.
-            metrics: { selected: selected + (child.summary?.selected ?? 0), acted: acted + (child.summary?.acted ?? 0) },
+            metrics: {
+              selected: selected + (child.summary?.selected ?? 0),
+              acted: acted + (child.summary?.acted ?? 0),
+              ...(unmeasured || child.summary?.unmeasured ? { unmeasuredEffect: true } : {}),
+            },
           };
         }
         // Synchronous completion — record and advance.
         state.started = idx + 1;
         state.results.push(child.output ?? null);
         selected += child.summary?.selected ?? 0;
         acted += child.summary?.acted ?? 0;
+        // One uncountable effect anywhere in the batch makes the batch's
+        // `acted` incomplete — the flag rides out with this entry's metrics.
+        if (child.summary?.unmeasured) unmeasured = true;
       }
 
       // All items done.
@@ -199,7 +210,7 @@ export function registerMapNode(engine: AutomationEngine, ctx: PluginContext): v
       return {
         success: true,
         output: { results: state.results, count: state.results.length },
-        metrics: { selected, acted },
+        metrics: { selected, acted, ...(unmeasured ? { unmeasuredEffect: true } : {}) },
       };
     },
   });