feat(spec,service-automation): surface per-run flow summaries — selected / acted / skipped (#4354) - #4377
Merged
Merged
Conversation
…ted / acted / skipped (#4354) `success: true` never meant "it did its job". A scheduled sweep that selects thirty records and writes none is, from outside, identical to one with nothing to do: same green status, same empty output, same silence, same schedule tomorrow. Nothing anywhere separated "nothing to do" from "broken" — which is how #4347 left three hotcrm production flows completely inert, running daily and green for as long as they had existed. Every terminal run now carries a `FlowRunSummary`: `selected` / `acted` / `skipped` totals, a per-node breakdown, and which gates closed. It is surfaced on `AutomationResult`, in `listRuns` / `getRun`, as one greppable log line, and as queryable `selected_count` / `acted_count` / `skipped_count` columns on `sys_automation_run` — so `selected > 0 AND acted = 0` over consecutive runs is something an operator can alert on rather than notice. The counts are declared by the executors (`NodeExecutionResult.metrics`), not sniffed from node output: only the node knows whether its result is a row count, a record, or a boolean. A closed conditional edge now records a `skipped` step naming the gate — the event that previously left no trace at all — and that step is explicitly not a run, so the ADR-0044 re-entry guard and the per-node `runs` counts are unchanged. Two details decide whether the numbers can be trusted: the summary is folded from the full step log before history compaction (and rehydrated from `summary_json`, never re-folded from the compacted steps), and `subflow` / `map` children roll up into their parent — synchronously via `metrics`, and via `creditChildRun` when the child paused and bubbles back. Additive throughout: `summary` is optional everywhere, legacy rows keep `null` counts rather than `0`, and no execution behaviour changes. The log line defaults to `info`, with `runSummaryLog: 'debug' | 'off'` to lower the volume without turning the measurement off. Verified: service-automation 525/46 (26 new), spec 7165/279 (5 new), runtime 974/68, plugin-approvals 330/13; all 8 spec `check:generated` gates plus `check:liveness` and `check:exported-any`; eslint clean; `tsc --noEmit` at the ledgered 2 pre-existing errors. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzJqHp1yXjri1tqKP8WoL4
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
github-actions
Bot
added
documentation
Contributor
📓 Docs Drift CheckThis PR changes 2 package(s): 107 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
|
…ness-ledger count (#4354) Two CI gates on the previous commit, both real: - `check:nul-bytes` — the gate-dedup key joined its parts with LITERAL NUL bytes instead of the escape sequence. Byte-identical at runtime, but a raw NUL makes ripgrep treat the whole file as binary and silently return zero matches, so `run-summary.ts` would have dropped out of code search and every grep-based lint. Written as the escape now, with the reason the joiner is NUL at all spelled out (an edge id is author-supplied text, so a printable separator could collide two distinct gates). - `check:strictness-ledger` — `automation/execution.zod.ts` gained 5 `z.object(` sites (the run-summary family), so its ledger row and the section total were stale. Re-read: they are engine-emitted telemetry that nobody authors, so the row's `wire` verdict covers them unchanged; count updated to 13 (98 for the section) with the additions named. Verified: service-automation 525/46; `pnpm lint`, `check:nul-bytes` and `check:slot-lookup` clean; all seven pure spec audits (liveness, empty-state, variant-docs, strictness-ledger, react-conformance, skill-examples, exported-any) green. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_01SzJqHp1yXjri1tqKP8WoL4
os-zhuang
marked this pull request as ready for review
July 31, 2026 10:29
os-zhuang
added a commit
that referenced
this pull request
Jul 31, 2026
…ncomplete, instead of guessing (#4354) (#4397) #4354 shipped selected/acted counts sourced from the executors that know what they did, but left out the four node types a flow uses to act on anything OUTSIDE the platform. That gap was not cosmetic: a sweep whose whole job runs through a connector reported `acted: 0` and looked exactly like the dead sweep the counter exists to find. A detector that fires on healthy runs is worse than none — operators tune it out, and then it is not watching the flows that really did stop. Closing it needed a third answer, because for two of those nodes the platform genuinely cannot know what happened: - `connector_action` — `ConnectorActionDescriptor` declares nothing about whether an action reads or writes, so `acted: 0` understates a create and `acted: 1` overstates a lookup (and makes the alert never fire, which is the original bug one layer out). Reports `unmeasuredEffect` instead. #4395 proposes declaring the effect kind, which would make it a real count. - `http` — knowable from the method. GET/HEAD/OPTIONS report a real `acted: 0`; an accepted mutating call reports 1; `durable: true` reports 1 (the outbox row is a durable effect this run caused); a rejected or timed-out mutating call reports unmeasured, because a 500 can arrive after the write landed. - `script` — deliberately unchanged. A registered function is contractually pure (data I/O stays on the flow graph), so reporting no record metrics is accurate rather than a guess. Nothing enforces that purity — filed as #4396 rather than papered over, since a blanket `unmeasuredEffect` here would suppress the signal on every flow calling any function to cover one contract violation. The alert gains a clause: `selected > 0 AND acted = 0 AND unmeasured = 0`, with an `unmeasured_count` column to serve it — without the third clause it fires on every healthy connector-driven flow. The log line gains `unmeasured=N` only when non-zero, since its PRESENCE is what a reader must not miss. `unmeasured` propagates through subflow/map roll-ups and `creditChildRun`, so a parent whose child dispatched an uncountable effect knows its own `acted` is incomplete. `FlowRunSummary.unmeasured` is optional and undefined is NOT 0: a run recorded before this existed did not track uncountable effects at all. Verified: service-automation 546/47 (21 new), spec 7193/281 (2 new); all 8 check:generated gates plus the seven pure audits; check:nul-bytes and eslint clean. Branch restarted from main after #4377 merged; #4347's fix confirmed first by running that issue's own repro. Co-authored-by: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com> Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4354.
问题
success: true从来不代表"它干活了"。一个定时 flow 选出 30 条记录、然后一条也没写 —— 从外部看,和"本来就没活可干"完全一样:同样绿、同样没输出、同样静默、明天同样按时再跑一次。平台里没有任何地方能区分这两种状态。这不是假设。#4347 让 hotcrm 的三条生产 flow 完全失效 —— stalled-deal sweep 找齐了每一条停滞商机却没提醒任何人,续约 sweep 一单没约,campaign 动作一个 lead 没入。它们每天准时跑、一直是绿的,只有靠"断言写入记录"的运行时测试才被发现。自动化恰恰是没人盯着的那一类:UI bug 一小时内就有工单,死掉的 sweep 什么都不产生,跑得越久,静默看起来越正常。
做了什么
每个终态运行现在都带一份
FlowRunSummary,四个面同时暴露:selectedactedskippednodes[]runs/failures/skipped+ 自己的 selected/actedgates[]暴露位置:
AutomationResult.summary、listRuns/getRun、每次终态运行一行结构化日志、以及sys_automation_run上的可查询列。三个关键取舍
计数由执行器申报,不由引擎嗅探。 执行器返回
NodeExecutionResult.metrics,因为只有节点知道自己的result是什么意思:update_record批量写返回行数、按 id 写返回记录,delete_record可能返回布尔,notify返回投递数。引擎去猜这些形状就是在猜,而"会猜的机器可读接口"比没有更糟(ADR-0076 D12)。不碰记录的节点(decision、assignment)完全不上报 —— 缺失 ≠ 0。gate 被点名。 条件出边判为 false 时,现在会记一条
skippedstep,带上关掉它的 gate。这个事件此前完全没有痕迹,正是 #4347 隐形的原因。skipped step 明确不算一次运行:ADR-0044 重入守卫、每节点runs、节点状态都排除它 —— 新的可观测信号绝不能改变执行语义。可查询,所以能告警而不是靠"发现"。
sys_automation_run新增selected_count/acted_count/skipped_count列 +summary_json明细:用列而不是塞进 JSON:运维只能对"可过滤的东西"告警,这是仪表盘和警报器的区别。此前写入的行是
null而非0—— "没测量"不能读成"测量结果是零",否则第一次写这条查询时每一条历史行都是误报。两个决定数字可信度的细节
summary_json而不是重新折叠被压缩过的 steps —— 否则 5000 轮的 sweep 会报成只写了两百来条。subflow/map子运行会上卷到父运行:同步完成走metrics,子运行 pause 过再冒泡回来的走creditChildRun(父节点那一步是在挂起时写的,当时子运行还什么都没做)。没有这个,把写操作全都委托出去的 sweep 会报acted: 0—— 健康的运行触发本该抓坏运行的探测器,比没信号更糟。兼容性
全程增量:
summary处处可选,老的运行和 store 照常工作,执行行为零变化。日志行默认info(在生产日志级别看不见的行,和本 issue 要消除的"无信号"是一回事),高频 flow 可用AutomationServicePlugin的runSummaryLog: 'debug' | 'off'调低音量 —— 那是关于噪音的决定,不是关于要不要测量的决定。不修 #4347 本身 —— 这是本该抓到它的那把尺子。
新增 spec 导出
FlowRunSummarySchema、FlowRunNodeSummarySchema、FlowRunGateSummarySchema、ExecutionStepMetricsSchema、ExecutionStepSkipReasonSchema(及推导类型);ExecutionLog.summary、ExecutionStepLog.metrics/.skippedBy。service-automation导出summarizeRun/formatRunSummaryLine,让自建可观测面的宿主复用平台定义而不是各写一套。验证
@objectstack/service-automation525 / 46(新增 26)@objectstack/spec7165 / 279(新增 5)@objectstack/runtime974 / 68、@objectstack/plugin-approvals330 / 13check:generated全绿,外加check:liveness、check:exported-anytsc --noEmit停在 ledger 记录的 2 个既有错误上(未增加)🤖 Generated with Claude Code
https://claude.ai/code/session_01SzJqHp1yXjri1tqKP8WoL4
Generated by Claude Code