feat(spec,metadata-protocol,metadata,objectql,service-automation): stored metadata replays the full conversion chain at rehydration (#3903) (#4317)

The spec contract stopped at authored source: sys_metadata rows were
rehydrated unparsed and unconverted, so the authored and stored contracts
silently diverged (option 1 of #3903, chosen because #3855 already concluded
operator-run migrations cannot be relied on, and out-of-scope-by-policy would
fossilize the accommodations PD #12 names as debt).

- spec: new `applyConversionsToStoredItem(type, item, options?)` — one
  primitive encoding the data-at-rest policy: the FULL chain replays,
  retiredFromLoadPath entries included, because retirement is an
  authoring-surface event and a row at rest has no author to teach.
- metadata-protocol: wired at loadMetaFromDb, getMetaItems (active+drafts),
  getMetaItem (active+draft), getMetaItemLayered, duplicatePackage. Flow rows
  are deliberately skipped (the flow conversions' open-namespace conflict
  guard needs the automation engine's executor registry).
- metadata: DatabaseLoader live-row reads convert; history stays verbatim
  (the version layer records what was written — checksums must keep pairing).
- objectql: authored-action/-hook table reads convert (a stored pre-17 action
  row's `execute` dispatches via `target` again); the rule-validator's
  `requiredWhen ?? conditionalRequired` fallback — kept in #3883 with a
  retirement promise that had no mechanism — is deleted, its job done by the
  chain at every seam.
- service-automation: registerFlow passes includeRetired, so stored flows
  keep canonicalizing after their conversions graduate out of the window.
- registry.ts swallow answered (#3903 closing note): loadMetaFromDb validates
  the CONVERTED row, returns {loaded, errors, invalid}, and warns with a
  stable [metadata_spec_invalid] marker; registration proceeds deliberately
  (refusal at boot unhooks live tables and makes the row unfixable in
  Studio) — the enforcing gates are saveMetaItem's 422 and _diagnostics on
  read, and the registerItem hook is now documented as the diagnostic it is.

Policy written down in the ADR-0087 addendum ("stored metadata replays the
chain") and PD #12.

Closes #3903.

Co-authored-by: Jack Zhuang <277994282+os-zhuang@users.noreply.github.com>
Co-authored-by: Claude Fable 5 <noreply@anthropic.com>

Author: os-zhuang

.changeset/stored-metadata-replays-the-chain.md

@@ -0,0 +1,59 @@
+---
+"@objectstack/spec": minor
+"@objectstack/metadata-protocol": patch
+"@objectstack/metadata": patch
+"@objectstack/objectql": patch
+"@objectstack/service-automation": patch
+---
+
+feat(spec,metadata-protocol,metadata,objectql,service-automation): stored metadata replays the full conversion chain at rehydration (#3903)
+
+Every mechanism the platform has for evolving the metadata contract — schema
+transforms, the ADR-0087 D2 conversion layer, the D3 migration chain, the
+protocol-17 tombstones — operated on **authored source** only. Metadata **at
+rest** (`sys_metadata` rows written by Studio or the runtime authoring APIs)
+was rehydrated unparsed and unconverted, so the authored and stored contracts
+silently diverged: a pre-17 row carrying `conditionalRequired` or `execute`
+read as whatever each ad-hoc consumer happened to do with it.
+
+**New spec primitive — `applyConversionsToStoredItem(type, item, options?)`**
+(exported from the package root). Wraps one stored item of a given metadata
+type and replays the **full** conversion chain over it — `retiredFromLoadPath`
+entries included, because retirement is an *authoring-surface* event: the
+window exists to teach a live author, and a row at rest has no author to
+teach. Idempotent, never throws, never validates.
+
+Wired at every stored-row rehydration seam:
+
+- `metadata-protocol`: `loadMetaFromDb`, `getMetaItems` (active + draft
+  preview), `getMetaItem` (active + draft), `getMetaItemLayered`, and
+  `duplicatePackage` (a copy re-saves through the schema gate, so legacy
+  sources now duplicate successfully — and the copy is canonical).
+- `metadata`: the DatabaseLoader's live-row reads (`load` / `loadMany`).
+  History reads stay verbatim — history records what was written.
+- `objectql`: the authored-action / authored-hook direct table reads, so
+  runtime-authored actions stored with the removed `execute` alias dispatch
+  via `target` again.
+- `service-automation`: `AutomationEngine.registerFlow` now passes
+  `includeRetired` — stored flows keep canonicalizing after their conversions
+  graduate out of the load window. (The generic metadata seams deliberately
+  skip `type: 'flow'`: flow conversions carry the open-namespace conflict
+  guard, which needs this engine's live executor registry.)
+
+**Boot hydration diagnoses instead of shrugging.** `loadMetaFromDb` now
+returns `{ loaded, errors, invalid }`: each row is validated against its
+type's spec schema *after* conversion, and a genuine contract violation is
+counted and warned with a stable `[metadata_spec_invalid]` marker — but still
+registered, deliberately: refusing at boot would unhook live tables and make
+the row unlistable and unfixable in Studio. The write path (`saveMetaItem` →
+422) and the read-side `_diagnostics` envelope remain the enforcing gates; the
+`SchemaRegistry.registerItem` validation hook is now documented as exactly
+that diagnostic.
+
+**Retired accommodation.** With the chain running on every stored read path,
+the rule-validator's `requiredWhen ?? conditionalRequired` fallback — kept in
+#3883 with a retirement promise that had no mechanism — is deleted. If you
+call `evaluateValidationRules` directly with raw legacy field definitions,
+convert them first (`applyConversionsToStoredItem('object', def)`) or author
+`requiredWhen`; the platform's own read paths already hand you canonical
+shapes.

AGENTS.md

@@ -74,7 +74,7 @@ Other scripts: `objectui:bump` (pull only), `objectui:build`, `objectui:clean`.
    When renaming a legacy var, use `readEnvWithDeprecation('OS_NEW', 'LEGACY')` from `@objectstack/types` (keeps legacy working one release). Third-party exceptions kept as-is: `NODE_ENV`, `HOME`, `OPENAI_API_KEY`, `TURSO_*`, OAuth `*_CLIENT_ID/SECRET`, `RESEND_API_KEY`, `POSTMARK_TOKEN`, `AI_GATEWAY_*`, `SMTP_*`. See #1382.
 10. **File issues for out-of-scope findings — don't silently expand scope or leave them buried.** When you hit a bug, gap, or unenforced capability that's unrelated to the current task, or too large to fix in scope, open a GitHub issue (`gh issue create`) with a clear repro/decision and link it from your PR. Corollary: **never advertise or demo a capability the runtime doesn't actually deliver** (declared ≠ enforced) — fix it, trim it, or file an issue, but don't fake coverage. Example: the spec once declared 9 validation-rule types while the write-path validator enforced only 3 (`state_machine`/`script`/`cross_field`); the gap was filed as #1475 rather than demoed in the showcase, then closed by **trimming** what could never be enforced (`unique`/`async`/`custom`) and **implementing** the rest — the spec now declares 6 and `rule-validator.ts` handles all 6. Note how narrow that claim stayed even so: the evaluator was wired into insert and single-id update only, so a bulk `updateMany` silently skipped every rule — a second `declared ≠ enforced` gap one layer down, at the **call site** rather than the `switch`; filed as #3106 and closed by evaluating the bulk match set per row. A `case` label is not enforcement; check the **call site**.
 11. **Worktree-first — never edit on the shared `main` checkout.** This repo is edited by **multiple agents at once**; the shared `main` tree has its HEAD switched and reset *under you*, silently clobbering uncommitted work. Before your **first file edit**, you MUST be in a dedicated worktree on a feature branch: `git worktree add ../objectstack-<task> -b <branch> main && cd ../objectstack-<task> && pnpm install`. A PreToolUse hook (`.claude/hooks/guard-main-checkout.sh`) **enforces** this — it blocks `Edit`/`Write`/`NotebookEdit` unless the edited file is in a dedicated **worktree** — a feature branch on the *shared* checkout is **not** enough (it still gets switched under you) — and it checks the **edited file's own repo**, so sibling repos (`objectui`/`cloud`) you touch are covered too (override for a deliberate non-task fix with `OS_ALLOW_MAIN_EDITS=1`). Full playbook below.
-12. **Contract-first — fix the metadata, not the runtime.** This is a metadata-driven framework: `packages/spec` is the one contract between metadata *producers* and the runtime/renderers that *consume* it. When a piece of metadata "doesn't work," ask **first**: *is it spec-compliant? is this the long-term-correct direction?* If the metadata is wrong, fix it at the **producer** and **reject it at authoring/publish** (validation / lint) so the error surfaces loudly — do **not** add a lenient alias or `??` fallback in the consumer (a node executor, the REST layer, a renderer) to tolerate off-spec input. A tolerant fallback fossilizes the wrong convention into a second de-facto contract, dilutes the spec, and hides the producer's bug — one strict contract beats N dialects. This is an **internal** contract (we own both ends), so "be liberal in what you accept" (Postel) does **not** apply — that's for untrusted boundaries. Change the **spec** only when the spec itself is genuinely wrong, and then deliberately (edit the Zod schema + migrate), never by accreting consumer-side fallbacks. The `cfg.filter ?? cfg.filters` / `cfg.objectName ?? cfg.object` fallbacks the flow executors once carried are **debt to pay down, not a pattern to copy** — and the way they are being paid down is the pattern to copy. `filters` → `filter` has **graduated** into the ADR-0087 D2 conversion layer (`flow-node-crud-filter-alias`): rewritten to the canonical key at load, including the `AutomationEngine.registerFlow` rehydration seam, so the CRUD executors read `cfg.filter` directly and no consumer-side fallback survives. `object` → `objectName` and the six open-coded stragglers #3796 tracked (notify `to`/`subject`/`body`/`url`, script `functionName`/`input`) graduated the same way at protocol 17 (`flow-node-crud-object-alias`, `flow-node-notify-config-aliases`, `flow-node-script-config-aliases`), emptying the `readAliasedConfig` executor shim — deleted with them. When you must tolerate an alias at all, declare it as a conversion-layer entry (never a bare `??`, and no new executor shims) so it is declared, loud, tested, and *removable on a schedule*. *Worked example:* an AI-authored `create_record` used `fieldValues` / `today()` / `{{trigger.record.id}}` while the executor reads `fields` / `{TODAY()}` / `{record.id}` → the fix was correcting the authoring skill + a publish-gate lint that rejects the wrong shape (cloud#688), **not** a `cfg.fields ?? cfg.fieldValues` runtime alias (framework#2419, rejected). Strengthens #5.
+12. **Contract-first — fix the metadata, not the runtime.** This is a metadata-driven framework: `packages/spec` is the one contract between metadata *producers* and the runtime/renderers that *consume* it. When a piece of metadata "doesn't work," ask **first**: *is it spec-compliant? is this the long-term-correct direction?* If the metadata is wrong, fix it at the **producer** and **reject it at authoring/publish** (validation / lint) so the error surfaces loudly — do **not** add a lenient alias or `??` fallback in the consumer (a node executor, the REST layer, a renderer) to tolerate off-spec input. A tolerant fallback fossilizes the wrong convention into a second de-facto contract, dilutes the spec, and hides the producer's bug — one strict contract beats N dialects. This is an **internal** contract (we own both ends), so "be liberal in what you accept" (Postel) does **not** apply — that's for untrusted boundaries. Change the **spec** only when the spec itself is genuinely wrong, and then deliberately (edit the Zod schema + migrate), never by accreting consumer-side fallbacks. The `cfg.filter ?? cfg.filters` / `cfg.objectName ?? cfg.object` fallbacks the flow executors once carried are **debt to pay down, not a pattern to copy** — and the way they are being paid down is the pattern to copy. `filters` → `filter` has **graduated** into the ADR-0087 D2 conversion layer (`flow-node-crud-filter-alias`): rewritten to the canonical key at load, including the `AutomationEngine.registerFlow` rehydration seam, so the CRUD executors read `cfg.filter` directly and no consumer-side fallback survives. `object` → `objectName` and the six open-coded stragglers #3796 tracked (notify `to`/`subject`/`body`/`url`, script `functionName`/`input`) graduated the same way at protocol 17 (`flow-node-crud-object-alias`, `flow-node-notify-config-aliases`, `flow-node-script-config-aliases`), emptying the `readAliasedConfig` executor shim — deleted with them. When you must tolerate an alias at all, declare it as a conversion-layer entry (never a bare `??`, and no new executor shims) so it is declared, loud, tested, and *removable on a schedule*. Stored `sys_metadata` rows (data at rest) are covered from the other side: every rehydration seam replays the **full** conversion chain — retired entries included — via `applyConversionsToStoredItem` (#3903, ADR-0087 addendum), so a consumer never needs its own accommodation for a legacy stored shape either. *Worked example:* an AI-authored `create_record` used `fieldValues` / `today()` / `{{trigger.record.id}}` while the executor reads `fields` / `{TODAY()}` / `{record.id}` → the fix was correcting the authoring skill + a publish-gate lint that rejects the wrong shape (cloud#688), **not** a `cfg.fields ?? cfg.fieldValues` runtime alias (framework#2419, rejected). Strengthens #5.
 
 ---
 

docs/adr/0087-metadata-protocol-upgrade-contract.md

@@ -381,3 +381,43 @@ does both halves:
 - **Backfilled history joins the registry, not the loader:** the protocol-11
   `compactLayout`→`highlightFields` rename (retired at authoring in 11.9.1,
   pre-dating this ADR) is also preserved as a retired step-11 conversion.
+
+## Addendum (2026-07-31) — stored metadata replays the chain (#3903)
+
+Everything above serves **authored source**: `normalizeStackInput` converts at
+`defineStack`/`validate`/`lint`, tombstones teach the author, `migrate meta`
+rewrites files. Metadata **at rest** — `sys_metadata` rows written by Studio or
+the runtime authoring APIs — was reached by none of it: rows were rehydrated
+unparsed and unconverted, so the authored and stored contracts silently
+diverged (#3903). This addendum extends the contract to data at rest:
+
+- **Every stored-row rehydration seam replays the FULL chain, retired entries
+  included** — `applyConversionsToStoredItem` in `spec/conversions/stored.ts`
+  is the one primitive, called by `loadMetaFromDb`, `getMetaItems` (active and
+  draft), `getMetaItem`, `getMetaItemLayered`, `duplicatePackage`, the
+  DatabaseLoader's live-row reads, and objectql's authored-action/-hook table
+  reads. Rationale: **retirement is an authoring-surface event.** The window
+  exists so a live author is taught the canonical spelling; a row at rest has
+  no author to teach, and refusing its historical shape would only break data
+  that once worked. D3 keeps every conversion forever precisely so any past
+  major replays forward — a stored row is the perpetual "consumer arriving
+  late", and the read path is its chain.
+- **Flows canonicalize at their own seam.** `AutomationEngine.registerFlow`
+  (the rehydration seam PD #12 names) now also replays retired entries; the
+  generic metadata seams deliberately skip `type: 'flow'` because flow-node
+  conversions carry the ADR-0078 open-namespace conflict guard, which needs
+  the engine's live executor registry (`reservedNodeTypes`).
+- **The version layer stays verbatim.** `sys_metadata_history` reads and
+  `SysMetadataRepository` bodies are NOT converted — history is a record of
+  what was written, and converting would break the checksum↔body pairing.
+  Conversion happens where rows become *served metadata*, not where versions
+  are stored.
+- **Writes stay gated; reads diagnose, never drop.** `saveMetaItem` keeps
+  rejecting off-spec bodies (422, tombstones included) — new rows are always
+  canonical, so the stored pass is a strictly shrinking concern. On the read
+  side, what still fails the current schema *after* conversion is a genuine
+  contract violation: `loadMetaFromDb` counts it (`invalid`), warns with a
+  stable `[metadata_spec_invalid]` marker, and registers it anyway — refusal
+  at boot would unhook live tables and make the row unfixable in Studio
+  (availability over purity for data at rest; the same verdict reaches Studio
+  as `_diagnostics` on every read).