diff --git a/.changeset/stored-metadata-replays-the-chain.md b/.changeset/stored-metadata-replays-the-chain.md new file mode 100644 index 0000000000..79aa0be13f --- /dev/null +++ b/.changeset/stored-metadata-replays-the-chain.md @@ -0,0 +1,59 @@ +--- +"@objectstack/spec": minor +"@objectstack/metadata-protocol": patch +"@objectstack/metadata": patch +"@objectstack/objectql": patch +"@objectstack/service-automation": patch +--- + +feat(spec,metadata-protocol,metadata,objectql,service-automation): stored metadata replays the full conversion chain at rehydration (#3903) + +Every mechanism the platform has for evolving the metadata contract — schema +transforms, the ADR-0087 D2 conversion layer, the D3 migration chain, the +protocol-17 tombstones — operated on **authored source** only. Metadata **at +rest** (`sys_metadata` rows written by Studio or the runtime authoring APIs) +was rehydrated unparsed and unconverted, so the authored and stored contracts +silently diverged: a pre-17 row carrying `conditionalRequired` or `execute` +read as whatever each ad-hoc consumer happened to do with it. + +**New spec primitive — `applyConversionsToStoredItem(type, item, options?)`** +(exported from the package root). Wraps one stored item of a given metadata +type and replays the **full** conversion chain over it — `retiredFromLoadPath` +entries included, because retirement is an *authoring-surface* event: the +window exists to teach a live author, and a row at rest has no author to +teach. Idempotent, never throws, never validates. + +Wired at every stored-row rehydration seam: + +- `metadata-protocol`: `loadMetaFromDb`, `getMetaItems` (active + draft + preview), `getMetaItem` (active + draft), `getMetaItemLayered`, and + `duplicatePackage` (a copy re-saves through the schema gate, so legacy + sources now duplicate successfully — and the copy is canonical). +- `metadata`: the DatabaseLoader's live-row reads (`load` / `loadMany`). + History reads stay verbatim — history records what was written. +- `objectql`: the authored-action / authored-hook direct table reads, so + runtime-authored actions stored with the removed `execute` alias dispatch + via `target` again. +- `service-automation`: `AutomationEngine.registerFlow` now passes + `includeRetired` — stored flows keep canonicalizing after their conversions + graduate out of the load window. (The generic metadata seams deliberately + skip `type: 'flow'`: flow conversions carry the open-namespace conflict + guard, which needs this engine's live executor registry.) + +**Boot hydration diagnoses instead of shrugging.** `loadMetaFromDb` now +returns `{ loaded, errors, invalid }`: each row is validated against its +type's spec schema *after* conversion, and a genuine contract violation is +counted and warned with a stable `[metadata_spec_invalid]` marker — but still +registered, deliberately: refusing at boot would unhook live tables and make +the row unlistable and unfixable in Studio. The write path (`saveMetaItem` → +422) and the read-side `_diagnostics` envelope remain the enforcing gates; the +`SchemaRegistry.registerItem` validation hook is now documented as exactly +that diagnostic. + +**Retired accommodation.** With the chain running on every stored read path, +the rule-validator's `requiredWhen ?? conditionalRequired` fallback — kept in +#3883 with a retirement promise that had no mechanism — is deleted. If you +call `evaluateValidationRules` directly with raw legacy field definitions, +convert them first (`applyConversionsToStoredItem('object', def)`) or author +`requiredWhen`; the platform's own read paths already hand you canonical +shapes. diff --git a/AGENTS.md b/AGENTS.md index fe531a76de..78bf1f35d1 100644 --- a/AGENTS.md +++ b/AGENTS.md @@ -74,7 +74,7 @@ Other scripts: `objectui:bump` (pull only), `objectui:build`, `objectui:clean`. When renaming a legacy var, use `readEnvWithDeprecation('OS_NEW', 'LEGACY')` from `@objectstack/types` (keeps legacy working one release). Third-party exceptions kept as-is: `NODE_ENV`, `HOME`, `OPENAI_API_KEY`, `TURSO_*`, OAuth `*_CLIENT_ID/SECRET`, `RESEND_API_KEY`, `POSTMARK_TOKEN`, `AI_GATEWAY_*`, `SMTP_*`. See #1382. 10. **File issues for out-of-scope findings — don't silently expand scope or leave them buried.** When you hit a bug, gap, or unenforced capability that's unrelated to the current task, or too large to fix in scope, open a GitHub issue (`gh issue create`) with a clear repro/decision and link it from your PR. Corollary: **never advertise or demo a capability the runtime doesn't actually deliver** (declared ≠ enforced) — fix it, trim it, or file an issue, but don't fake coverage. Example: the spec once declared 9 validation-rule types while the write-path validator enforced only 3 (`state_machine`/`script`/`cross_field`); the gap was filed as #1475 rather than demoed in the showcase, then closed by **trimming** what could never be enforced (`unique`/`async`/`custom`) and **implementing** the rest — the spec now declares 6 and `rule-validator.ts` handles all 6. Note how narrow that claim stayed even so: the evaluator was wired into insert and single-id update only, so a bulk `updateMany` silently skipped every rule — a second `declared ≠ enforced` gap one layer down, at the **call site** rather than the `switch`; filed as #3106 and closed by evaluating the bulk match set per row. A `case` label is not enforcement; check the **call site**. 11. **Worktree-first — never edit on the shared `main` checkout.** This repo is edited by **multiple agents at once**; the shared `main` tree has its HEAD switched and reset *under you*, silently clobbering uncommitted work. Before your **first file edit**, you MUST be in a dedicated worktree on a feature branch: `git worktree add ../objectstack- -b main && cd ../objectstack- && pnpm install`. A PreToolUse hook (`.claude/hooks/guard-main-checkout.sh`) **enforces** this — it blocks `Edit`/`Write`/`NotebookEdit` unless the edited file is in a dedicated **worktree** — a feature branch on the *shared* checkout is **not** enough (it still gets switched under you) — and it checks the **edited file's own repo**, so sibling repos (`objectui`/`cloud`) you touch are covered too (override for a deliberate non-task fix with `OS_ALLOW_MAIN_EDITS=1`). Full playbook below. -12. **Contract-first — fix the metadata, not the runtime.** This is a metadata-driven framework: `packages/spec` is the one contract between metadata *producers* and the runtime/renderers that *consume* it. When a piece of metadata "doesn't work," ask **first**: *is it spec-compliant? is this the long-term-correct direction?* If the metadata is wrong, fix it at the **producer** and **reject it at authoring/publish** (validation / lint) so the error surfaces loudly — do **not** add a lenient alias or `??` fallback in the consumer (a node executor, the REST layer, a renderer) to tolerate off-spec input. A tolerant fallback fossilizes the wrong convention into a second de-facto contract, dilutes the spec, and hides the producer's bug — one strict contract beats N dialects. This is an **internal** contract (we own both ends), so "be liberal in what you accept" (Postel) does **not** apply — that's for untrusted boundaries. Change the **spec** only when the spec itself is genuinely wrong, and then deliberately (edit the Zod schema + migrate), never by accreting consumer-side fallbacks. The `cfg.filter ?? cfg.filters` / `cfg.objectName ?? cfg.object` fallbacks the flow executors once carried are **debt to pay down, not a pattern to copy** — and the way they are being paid down is the pattern to copy. `filters` → `filter` has **graduated** into the ADR-0087 D2 conversion layer (`flow-node-crud-filter-alias`): rewritten to the canonical key at load, including the `AutomationEngine.registerFlow` rehydration seam, so the CRUD executors read `cfg.filter` directly and no consumer-side fallback survives. `object` → `objectName` and the six open-coded stragglers #3796 tracked (notify `to`/`subject`/`body`/`url`, script `functionName`/`input`) graduated the same way at protocol 17 (`flow-node-crud-object-alias`, `flow-node-notify-config-aliases`, `flow-node-script-config-aliases`), emptying the `readAliasedConfig` executor shim — deleted with them. When you must tolerate an alias at all, declare it as a conversion-layer entry (never a bare `??`, and no new executor shims) so it is declared, loud, tested, and *removable on a schedule*. *Worked example:* an AI-authored `create_record` used `fieldValues` / `today()` / `{{trigger.record.id}}` while the executor reads `fields` / `{TODAY()}` / `{record.id}` → the fix was correcting the authoring skill + a publish-gate lint that rejects the wrong shape (cloud#688), **not** a `cfg.fields ?? cfg.fieldValues` runtime alias (framework#2419, rejected). Strengthens #5. +12. **Contract-first — fix the metadata, not the runtime.** This is a metadata-driven framework: `packages/spec` is the one contract between metadata *producers* and the runtime/renderers that *consume* it. When a piece of metadata "doesn't work," ask **first**: *is it spec-compliant? is this the long-term-correct direction?* If the metadata is wrong, fix it at the **producer** and **reject it at authoring/publish** (validation / lint) so the error surfaces loudly — do **not** add a lenient alias or `??` fallback in the consumer (a node executor, the REST layer, a renderer) to tolerate off-spec input. A tolerant fallback fossilizes the wrong convention into a second de-facto contract, dilutes the spec, and hides the producer's bug — one strict contract beats N dialects. This is an **internal** contract (we own both ends), so "be liberal in what you accept" (Postel) does **not** apply — that's for untrusted boundaries. Change the **spec** only when the spec itself is genuinely wrong, and then deliberately (edit the Zod schema + migrate), never by accreting consumer-side fallbacks. The `cfg.filter ?? cfg.filters` / `cfg.objectName ?? cfg.object` fallbacks the flow executors once carried are **debt to pay down, not a pattern to copy** — and the way they are being paid down is the pattern to copy. `filters` → `filter` has **graduated** into the ADR-0087 D2 conversion layer (`flow-node-crud-filter-alias`): rewritten to the canonical key at load, including the `AutomationEngine.registerFlow` rehydration seam, so the CRUD executors read `cfg.filter` directly and no consumer-side fallback survives. `object` → `objectName` and the six open-coded stragglers #3796 tracked (notify `to`/`subject`/`body`/`url`, script `functionName`/`input`) graduated the same way at protocol 17 (`flow-node-crud-object-alias`, `flow-node-notify-config-aliases`, `flow-node-script-config-aliases`), emptying the `readAliasedConfig` executor shim — deleted with them. When you must tolerate an alias at all, declare it as a conversion-layer entry (never a bare `??`, and no new executor shims) so it is declared, loud, tested, and *removable on a schedule*. Stored `sys_metadata` rows (data at rest) are covered from the other side: every rehydration seam replays the **full** conversion chain — retired entries included — via `applyConversionsToStoredItem` (#3903, ADR-0087 addendum), so a consumer never needs its own accommodation for a legacy stored shape either. *Worked example:* an AI-authored `create_record` used `fieldValues` / `today()` / `{{trigger.record.id}}` while the executor reads `fields` / `{TODAY()}` / `{record.id}` → the fix was correcting the authoring skill + a publish-gate lint that rejects the wrong shape (cloud#688), **not** a `cfg.fields ?? cfg.fieldValues` runtime alias (framework#2419, rejected). Strengthens #5. --- diff --git a/docs/adr/0087-metadata-protocol-upgrade-contract.md b/docs/adr/0087-metadata-protocol-upgrade-contract.md index 80fcddb68f..d11fe813d2 100644 --- a/docs/adr/0087-metadata-protocol-upgrade-contract.md +++ b/docs/adr/0087-metadata-protocol-upgrade-contract.md @@ -381,3 +381,43 @@ does both halves: - **Backfilled history joins the registry, not the loader:** the protocol-11 `compactLayout`→`highlightFields` rename (retired at authoring in 11.9.1, pre-dating this ADR) is also preserved as a retired step-11 conversion. + +## Addendum (2026-07-31) — stored metadata replays the chain (#3903) + +Everything above serves **authored source**: `normalizeStackInput` converts at +`defineStack`/`validate`/`lint`, tombstones teach the author, `migrate meta` +rewrites files. Metadata **at rest** — `sys_metadata` rows written by Studio or +the runtime authoring APIs — was reached by none of it: rows were rehydrated +unparsed and unconverted, so the authored and stored contracts silently +diverged (#3903). This addendum extends the contract to data at rest: + +- **Every stored-row rehydration seam replays the FULL chain, retired entries + included** — `applyConversionsToStoredItem` in `spec/conversions/stored.ts` + is the one primitive, called by `loadMetaFromDb`, `getMetaItems` (active and + draft), `getMetaItem`, `getMetaItemLayered`, `duplicatePackage`, the + DatabaseLoader's live-row reads, and objectql's authored-action/-hook table + reads. Rationale: **retirement is an authoring-surface event.** The window + exists so a live author is taught the canonical spelling; a row at rest has + no author to teach, and refusing its historical shape would only break data + that once worked. D3 keeps every conversion forever precisely so any past + major replays forward — a stored row is the perpetual "consumer arriving + late", and the read path is its chain. +- **Flows canonicalize at their own seam.** `AutomationEngine.registerFlow` + (the rehydration seam PD #12 names) now also replays retired entries; the + generic metadata seams deliberately skip `type: 'flow'` because flow-node + conversions carry the ADR-0078 open-namespace conflict guard, which needs + the engine's live executor registry (`reservedNodeTypes`). +- **The version layer stays verbatim.** `sys_metadata_history` reads and + `SysMetadataRepository` bodies are NOT converted — history is a record of + what was written, and converting would break the checksum↔body pairing. + Conversion happens where rows become *served metadata*, not where versions + are stored. +- **Writes stay gated; reads diagnose, never drop.** `saveMetaItem` keeps + rejecting off-spec bodies (422, tombstones included) — new rows are always + canonical, so the stored pass is a strictly shrinking concern. On the read + side, what still fails the current schema *after* conversion is a genuine + contract violation: `loadMetaFromDb` counts it (`invalid`), warns with a + stable `[metadata_spec_invalid]` marker, and registers it anyway — refusal + at boot would unhook live tables and make the row unfixable in Studio + (availability over purity for data at rest; the same verdict reaches Studio + as `_diagnostics` on every read). diff --git a/packages/metadata-protocol/src/protocol.stored-conversions.test.ts b/packages/metadata-protocol/src/protocol.stored-conversions.test.ts new file mode 100644 index 0000000000..7fc3f6d0f0 --- /dev/null +++ b/packages/metadata-protocol/src/protocol.stored-conversions.test.ts @@ -0,0 +1,172 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #3903 — stored `sys_metadata` rows replay the FULL ADR-0087 conversion + * chain at rehydration. + * + * Every seam that turns a row's `metadata` JSON into an in-memory item + * (`loadMetaFromDb`, `getMetaItems`, `getMetaItem`) canonicalizes it first — + * including `retiredFromLoadPath` entries, because a row at rest has no + * author for a tombstone to teach: the alias that protocol 17 removed from + * the AUTHORING surface must keep reading canonically from data written + * under protocol ≤ 16 forever. + * + * The rows below are seeded directly into the stub engine — deliberately + * bypassing `saveMetaItem`'s schema gate, exactly like a real row written + * years ago under an older protocol. + */ +import { describe, expect, it } from 'vitest'; +import { ObjectStackProtocolImplementation } from './protocol.js'; + +interface Row { + id: string; + type: string; + name: string; + organization_id: string | null; + package_id: string | null; + state: string; + metadata: string; +} + +function matches(r: Row, where: Record): boolean { + for (const [k, v] of Object.entries(where)) { + if (v === undefined) continue; + if ((r as any)[k] !== v) return false; + } + return true; +} + +function makeStubEngine(seedRows: Array & { type: string; name: string; metadata: unknown }>) { + let nextId = 0; + const rows: Row[] = seedRows.map((r) => ({ + id: `r_${++nextId}`, + organization_id: null, + package_id: null, + state: 'active', + ...r, + metadata: typeof r.metadata === 'string' ? r.metadata : JSON.stringify(r.metadata), + })); + const registered: Array<{ kind: 'object' | 'item'; type?: string; body: any }> = []; + + const engine: any = { + async findOne(_t: string, opts: { where: Record }) { + return rows.find((r) => matches(r, opts.where)) ?? null; + }, + async find(_t: string, opts: { where: Record }) { + return rows.filter((r) => matches(r, opts.where)); + }, + async insert() { return { id: 'x' }; }, + async update() { return { id: 'x' }; }, + async delete() { return { deleted: 0 }; }, + registry: { + listItems: () => [], + isPackageDisabled: () => false, + registerItem: (type: string, body: any) => { registered.push({ kind: 'item', type, body }); }, + registerObject: (body: any) => { registered.push({ kind: 'object', body }); }, + }, + }; + return { engine, rows, registered }; +} + +// A protocol-≤16 object row: the `conditionalRequired` alias was removed from +// the spec in 17 (#3855) and its conversion is `retiredFromLoadPath` — the +// authored load seam refuses it, but the stored seam must keep lowering it. +const legacyObjectRow = { + type: 'object', + name: 'crm_invoice', + metadata: { + name: 'crm_invoice', + label: 'Invoice', + fields: { + status: { type: 'select', label: 'Status' }, + amount: { type: 'currency', label: 'Amount', conditionalRequired: "record.status == 'sent'" }, + }, + }, +}; + +// A pre-17 standalone action row still carrying the removed `execute` alias. +const legacyActionRow = { + type: 'action', + name: 'convert', + metadata: { name: 'convert', label: 'Convert', type: 'script', object: 'crm_invoice', execute: 'convertHandler' }, +}; + +describe('getMetaItems — stored rows are served canonical (#3903)', () => { + it('lowers the retired conditionalRequired alias on a stored object row', async () => { + const { engine } = makeStubEngine([legacyObjectRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + const res = await protocol.getMetaItems({ type: 'object' }); + const obj = (res.items as any[]).find((i) => i.name === 'crm_invoice'); + expect(obj.fields.amount.requiredWhen).toBe("record.status == 'sent'"); + expect('conditionalRequired' in obj.fields.amount).toBe(false); + }); + + it('lowers the retired execute alias on a stored action row', async () => { + const { engine } = makeStubEngine([legacyActionRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + const res = await protocol.getMetaItems({ type: 'action' }); + const action = (res.items as any[]).find((i) => i.name === 'convert'); + expect(action.target).toBe('convertHandler'); + expect('execute' in action).toBe(false); + }); + + it('deliberately does NOT convert flow rows — that seam is registerFlow (conflict guard needs the executor registry)', async () => { + const legacyFlow = { + type: 'flow', + name: 'purge_flow', + metadata: { + name: 'purge_flow', + nodes: [{ id: 'n1', type: 'delete_record', config: { objectName: 'lead', filters: { status: 'stale' } } }], + }, + }; + const { engine } = makeStubEngine([legacyFlow]); + const protocol = new ObjectStackProtocolImplementation(engine); + const res = await protocol.getMetaItems({ type: 'flow' }); + const flow = (res.items as any[]).find((i) => i.name === 'purge_flow'); + expect(flow.nodes[0].config.filters).toEqual({ status: 'stale' }); + }); +}); + +describe('getMetaItem — single stored read is canonical (#3903)', () => { + it('returns the converted body with clean _diagnostics (chain-owned history is not "invalid")', async () => { + const { engine } = makeStubEngine([legacyObjectRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + const res: any = await protocol.getMetaItem({ type: 'object', name: 'crm_invoice' }); + expect(res.item.fields.amount.requiredWhen).toBe("record.status == 'sent'"); + // Pre-#3903 this row validated RAW: the tombstoned alias made every + // legacy row read as invalid metadata. Converted first, it is valid. + expect(res.item._diagnostics?.valid).toBe(true); + }); +}); + +describe('loadMetaFromDb — boot hydration converts, diagnoses, never drops (#3903)', () => { + it('registers the CONVERTED body and reports invalid: 0 for chain-owned history', async () => { + const { engine, registered } = makeStubEngine([legacyObjectRow, legacyActionRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + const res = await protocol.loadMetaFromDb(); + expect(res).toEqual({ loaded: 2, errors: 0, invalid: 0 }); + + const obj = registered.find((r) => r.kind === 'object')!; + expect(obj.body.fields.amount.requiredWhen).toBe("record.status == 'sent'"); + expect('conditionalRequired' in obj.body.fields.amount).toBe(false); + + const action = registered.find((r) => r.kind === 'item' && r.type === 'action')!; + expect(action.body.target).toBe('convertHandler'); + }); + + it('counts a genuinely off-contract row as invalid but STILL registers it (availability)', async () => { + const broken = { + type: 'object', + name: 'corrupt_thing', + // `fields` as a number cannot be owned by any conversion — a real + // contract violation, not chain history. + metadata: { name: 'corrupt_thing', label: 'Corrupt', fields: 42 }, + }; + const { engine, registered } = makeStubEngine([broken]); + const protocol = new ObjectStackProtocolImplementation(engine); + const res = await protocol.loadMetaFromDb(); + expect(res.loaded).toBe(1); + expect(res.invalid).toBe(1); + expect(registered.some((r) => r.kind === 'object' && r.body?.name === 'corrupt_thing')).toBe(true); + }); +}); diff --git a/packages/metadata-protocol/src/protocol.ts b/packages/metadata-protocol/src/protocol.ts index ce4c6c70c5..2c57fc762f 100644 --- a/packages/metadata-protocol/src/protocol.ts +++ b/packages/metadata-protocol/src/protocol.ts @@ -20,6 +20,7 @@ import type { MetadataCacheRequest, MetadataCacheResponse, ServiceInfo, ApiRoute import { readServiceSelfInfo } from '@objectstack/spec/api'; import { parseFilterAST, isFilterAST, VALID_AST_OPERATORS, REFERENCE_VALUE_TYPES, type DroppedFieldsEvent, type QueryAST } from '@objectstack/spec/data'; import { PLURAL_TO_SINGULAR, SINGULAR_TO_PLURAL } from '@objectstack/spec/shared'; +import { applyConversionsToStoredItem } from '@objectstack/spec'; import { type FormView, isAggregatedViewContainer } from '@objectstack/spec/ui'; import { METADATA_FORM_REGISTRY, CORE_SERVICE_PROVIDER, serviceUnavailableMessage } from '@objectstack/spec/system'; import { DEFAULT_METADATA_TYPE_REGISTRY, getMetadataTypeSchema, getMetadataTypeActions, getMetadataCreateSeed } from '@objectstack/spec/kernel'; @@ -1543,6 +1544,47 @@ export class ObjectStackProtocolImplementation implements */ private authoringGates = new Map(); + /** + * Once-per-process dedupe for stored-row conversion notices + * (`conversionId|type|name`). `getMetaItems`/`getMetaItem` re-read + * sys_metadata on every call, so without this a single legacy row would + * warn on every list request instead of once. + */ + private storedConversionWarned = new Set(); + + /** + * Canonicalize a stored `sys_metadata` body on rehydration (#3903; + * ADR-0087 addendum "stored metadata replays the chain"). + * + * Every seam that turns a row's `metadata` JSON into an in-memory item + * funnels through here, so a row written under a past protocol is read + * in today's canonical shape — parity with what the authored load path + * has always done, extended by the full-chain replay data at rest needs + * (a stored row has no author for a tombstone to teach). + * + * `flow` is deliberately skipped: flow-node conversions carry an + * open-namespace conflict guard that needs the automation engine's live + * executor registry (`reservedNodeTypes`), which this layer does not + * have. Flows canonicalize at `AutomationEngine.registerFlow` — the + * execution seam — with the same full-chain policy. + */ + private convertStoredItem(type: string, data: unknown): unknown { + const singular = PLURAL_TO_SINGULAR[type] ?? type; + if (singular === 'flow') return data; + return applyConversionsToStoredItem(singular, data, { + onNotice: (n) => { + const name = (data as { name?: unknown } | null | undefined)?.name; + const key = `${n.conversionId}|${singular}|${String(name ?? '')}`; + if (this.storedConversionWarned.has(key)) return; + this.storedConversionWarned.add(key); + console.warn( + `[Protocol] stored ${singular}/${String(name ?? '')} carries a pre-protocol shape; ` + + `${n.message} The row itself is unchanged — re-save it (Studio edit → save) to persist the canonical shape.`, + ); + }, + }); + } + constructor( engine: IDataEngine, getServicesRegistry?: () => Map, @@ -2297,14 +2339,19 @@ export class ObjectStackProtocolImplementation implements const records = Array.from(mergedMap.values()); if (records && records.length > 0) { const isView = (PLURAL_TO_SINGULAR[request.type] ?? request.type) === 'view'; - // Parse each overlay body once and surface its persisted + // Parse each overlay body once — replaying the stored-row + // conversion chain (#3903) so every consumer of this list sees + // the canonical protocol shape — and surface its persisted // software-package binding so the sidebar package filter and // provenance classification see overlay rows the way they see // registry items. const overlays = records.map((record) => { - const data = typeof record.metadata === 'string' - ? JSON.parse(record.metadata) - : record.metadata; + const data = this.convertStoredItem( + String(record.type ?? request.type), + typeof record.metadata === 'string' + ? JSON.parse(record.metadata) + : record.metadata, + ) as any; const recPkg = (record as { package_id?: string | null }).package_id ?? undefined; if (recPkg && data && typeof data === 'object' && (data as any)._packageId === undefined) { (data as any)._packageId = recPkg; @@ -2388,7 +2435,10 @@ export class ObjectStackProtocolImplementation implements // previews only its own package's entry, so two packages' // same-name drafts stay distinct. Draft rows win over active. const drafts = draftRecords.map((record) => { - const data = typeof record.metadata === 'string' ? JSON.parse(record.metadata) : record.metadata; + const data = this.convertStoredItem( + String(record.type ?? request.type), + typeof record.metadata === 'string' ? JSON.parse(record.metadata) : record.metadata, + ) as any; const recPkg = (record as { package_id?: string | null }).package_id ?? undefined; if (recPkg && data && typeof data === 'object' && (data as any)._packageId === undefined) { (data as any)._packageId = recPkg; @@ -2557,9 +2607,12 @@ export class ObjectStackProtocolImplementation implements }; const draftRec = (orgId ? await findDraft(orgId) : undefined) ?? await findDraft(null); if (draftRec) { - const draftItem = typeof draftRec.metadata === 'string' - ? JSON.parse(draftRec.metadata) - : draftRec.metadata; + const draftItem = this.convertStoredItem( + String(draftRec.type ?? request.type), + typeof draftRec.metadata === 'string' + ? JSON.parse(draftRec.metadata) + : draftRec.metadata, + ) as any; if (draftItem && typeof draftItem === 'object') { const recPkg = (draftRec as { package_id?: string | null }).package_id ?? undefined; if (recPkg && (draftItem as any)._packageId === undefined) (draftItem as any)._packageId = recPkg; @@ -2614,9 +2667,12 @@ export class ObjectStackProtocolImplementation implements const record = (orgId ? await findOverlay(orgId) : undefined) ?? await findOverlay(null); if (record) { - item = typeof record.metadata === 'string' - ? JSON.parse(record.metadata) - : record.metadata; + item = this.convertStoredItem( + String(record.type ?? request.type), + typeof record.metadata === 'string' + ? JSON.parse(record.metadata) + : record.metadata, + ); // Surface the persisted software-package binding (parity with // the list path in getMetaItems) so provenance/UI can read it. const recPkg = (record as { package_id?: string | null }).package_id ?? undefined; @@ -2892,14 +2948,20 @@ export class ObjectStackProtocolImplementation implements if (orgId) { const rec = await findOverlay(orgId); if (rec) { - overlay = typeof rec.metadata === 'string' ? JSON.parse(rec.metadata) : rec.metadata; + overlay = this.convertStoredItem( + String(rec.type ?? request.type), + typeof rec.metadata === 'string' ? JSON.parse(rec.metadata) : rec.metadata, + ); overlayScope = 'org'; } } if (overlay === null) { const rec = await findOverlay(null); if (rec) { - overlay = typeof rec.metadata === 'string' ? JSON.parse(rec.metadata) : rec.metadata; + overlay = this.convertStoredItem( + String(rec.type ?? request.type), + typeof rec.metadata === 'string' ? JSON.parse(rec.metadata) : rec.metadata, + ); overlayScope = 'env'; } } @@ -6706,7 +6768,14 @@ export class ObjectStackProtocolImplementation implements const newName = renameName(row.name); let item: any; try { - item = typeof row.metadata === 'string' ? JSON.parse(row.metadata) : (row.metadata ?? {}); + // Canonicalize the source row before re-saving (#3903): the copy + // is a NEW write and must pass today's schema gate, so a legacy + // shape the chain owns is lifted rather than failing the copy — + // duplication never mints new rows in a pre-protocol dialect. + item = this.convertStoredItem( + String(row.type), + typeof row.metadata === 'string' ? JSON.parse(row.metadata) : (row.metadata ?? {}), + ); } catch { failed.push({ type: row.type, name: row.name, error: 'unparseable metadata' }); continue; @@ -7493,10 +7562,29 @@ export class ObjectStackProtocolImplementation implements * Per ADR-0005, project-kernel mode ALSO hydrates from sys_metadata — * customization overlay rows must survive restart. Scope filter * (`environment_id = this.environmentId ?? null`) keeps tenants isolated. + * + * #3903 — two contract duties run per row, and their split is deliberate: + * + * 1. **Convert** ({@link convertStoredItem}): the full ADR-0087 chain + * replays, so a row written under a past protocol registers in the + * canonical shape. Chain-owned history therefore stops presenting as + * "invalid metadata" at all. + * 2. **Diagnose, never drop**: what still fails the type's current spec + * schema *after* conversion is a genuine contract violation — counted + * in `invalid`, warned with a stable `[metadata_spec_invalid]` marker, + * and STILL registered. Boot-time refusal would unhook the metadata + * from every serving surface (an object row backs live tables; an + * unregistered item cannot even be listed, opened, or fixed in + * Studio), turning an upgrade into a data outage. The enforcing gates + * live where an author is present to act: `saveMetaItem` rejects new + * writes (422), and the read surfaces badge the row via + * `_diagnostics`. This is that same read-side verdict, surfaced once + * at boot where operators look. */ - async loadMetaFromDb(): Promise<{ loaded: number; errors: number }> { + async loadMetaFromDb(): Promise<{ loaded: number; errors: number; invalid: number }> { let loaded = 0; let errors = 0; + let invalid = 0; try { // ADR-0005 (revised 2026-05): hydrate only env-wide rows // (organization_id IS NULL). Per-org overlays are loaded on @@ -7509,11 +7597,26 @@ export class ObjectStackProtocolImplementation implements const records = await this.engine.find('sys_metadata', { where }); for (const record of records) { try { - const data = typeof record.metadata === 'string' - ? JSON.parse(record.metadata) - : record.metadata; + const data = this.convertStoredItem( + String(record.type), + typeof record.metadata === 'string' + ? JSON.parse(record.metadata) + : record.metadata, + ); // Normalize DB type to singular (DB may store legacy plural forms) const normalizedType = PLURAL_TO_SINGULAR[record.type] ?? record.type; + const verdict = computeMetadataDiagnostics(normalizedType, data); + if (verdict && !verdict.valid) { + invalid++; + const first = verdict.errors?.[0]; + console.warn( + `[Protocol] [metadata_spec_invalid] stored ${normalizedType}/${record.name} fails the ` + + `current spec schema even after conversion` + + (first ? ` (${first.path || ''}: ${first.message})` : '') + + `. Registered anyway so it stays serveable and fixable — correct it in Studio ` + + `(the read carries the full _diagnostics), or delete the sys_metadata row.`, + ); + } if (normalizedType === 'object') { this.engine.registry.registerObject(data as any, record.packageId || 'sys_metadata'); } else { @@ -7542,7 +7645,7 @@ export class ObjectStackProtocolImplementation implements console.warn(`[Protocol] DB hydration skipped: ${e.message}`); } } - return { loaded, errors }; + return { loaded, errors, invalid }; } // ========================================== diff --git a/packages/metadata/src/loaders/database-loader.ts b/packages/metadata/src/loaders/database-loader.ts index 598f88eb15..91b7c7c894 100644 --- a/packages/metadata/src/loaders/database-loader.ts +++ b/packages/metadata/src/loaders/database-loader.ts @@ -20,6 +20,8 @@ import type { MetadataHistoryRecord, } from '@objectstack/spec/system'; import { SysMetadataObject, SysMetadataHistoryObject } from '@objectstack/metadata-core'; +import { applyConversionsToStoredItem } from '@objectstack/spec'; +import { PLURAL_TO_SINGULAR } from '@objectstack/spec/shared'; import type { IDataDriver, IDataEngine } from '@objectstack/spec/contracts'; import type { MetadataLoader } from './loader-interface.js'; import { calculateChecksum } from '../utils/metadata-history-utils.js'; @@ -463,8 +465,24 @@ export class DatabaseLoader implements MetadataLoader { } /** - * Convert a database row to a metadata payload. - * Parses the JSON `metadata` column back into an object. + * Once-per-process dedupe for stored-row conversion notices — `load` / + * `loadMany` are hot read paths (cached, but re-hit on every TTL expiry), + * so a legacy row must warn once, not once per cache miss. + */ + private storedConversionWarned = new Set(); + + /** + * Convert a LIVE database row to a metadata payload. + * + * Parses the JSON `metadata` column back into an object, then replays the + * full ADR-0087 conversion chain over it (#3903): rows written under a past + * protocol are served canonical, exactly like the metadata-protocol's + * `sys_metadata` seams. History rows do NOT pass through here — history + * readers parse inline and stay verbatim, as a record of what was written. + * + * `flow` is skipped for the same reason the protocol skips it: flow-node + * conversions need the automation engine's live executor registry for their + * open-namespace conflict guard; flows canonicalize at `registerFlow`. */ private rowToData(row: Record): Record | null { if (!row || !row.metadata) return null; @@ -473,7 +491,18 @@ export class DatabaseLoader implements MetadataLoader { ? JSON.parse(row.metadata as string) : row.metadata; - return payload as Record; + const singular = PLURAL_TO_SINGULAR[row.type as string] ?? (row.type as string); + if (singular === 'flow') return payload as Record; + return applyConversionsToStoredItem(singular, payload as Record, { + onNotice: (n) => { + const key = `${n.conversionId}|${singular}|${String(row.name ?? '')}`; + if (this.storedConversionWarned.has(key)) return; + this.storedConversionWarned.add(key); + console.warn( + `[DatabaseLoader] stored ${singular}/${String(row.name ?? '')} carries a pre-protocol shape; ${n.message}`, + ); + }, + }); } /** diff --git a/packages/objectql/src/plugin.ts b/packages/objectql/src/plugin.ts index 7c69816b4c..115ec392ff 100644 --- a/packages/objectql/src/plugin.ts +++ b/packages/objectql/src/plugin.ts @@ -3,6 +3,7 @@ import { ObjectQL } from './engine.js'; import { assembleMetadataProtocol } from '@objectstack/metadata-protocol'; import { Plugin, PluginContext } from '@objectstack/core'; +import { applyConversionsToStoredItem } from '@objectstack/spec'; import { StorageNameMapping } from '@objectstack/spec/system'; import { LifecycleService } from './lifecycle/lifecycle-service.js'; import { lifecycleSettingsManifest } from './lifecycle/lifecycle-settings.js'; @@ -17,7 +18,7 @@ export type { Plugin, PluginContext }; * `@objectstack/spec`, since it is a server-side bootstrap concern only. */ interface ProtocolWithDbRestore { - loadMetaFromDb(): Promise<{ loaded: number; errors: number }>; + loadMetaFromDb(): Promise<{ loaded: number; errors: number; invalid?: number }>; } /** Type guard — checks whether the service exposes `loadMetaFromDb`. */ @@ -1056,10 +1057,13 @@ export class ObjectQLPlugin implements Plugin { // Phase 2: DB hydration (loads into SchemaRegistry) try { - const { loaded, errors } = await protocol.loadMetaFromDb(); + const { loaded, errors, invalid = 0 } = await protocol.loadMetaFromDb(); if (loaded > 0 || errors > 0) { - ctx.logger.info('Metadata restored from database to SchemaRegistry', { loaded, errors }); + // `invalid` (#3903): rows registered despite failing the current spec + // schema AFTER the stored conversion chain — each already warned with + // `[metadata_spec_invalid]` and carries `_diagnostics` on read. + ctx.logger.info('Metadata restored from database to SchemaRegistry', { loaded, errors, invalid }); } else { ctx.logger.debug('No persisted metadata found in database'); } @@ -1244,6 +1248,35 @@ export class ObjectQLPlugin implements Plugin { return registry.getArtifactItem('hook', name) !== undefined; } + /** + * Once-per-process dedupe for stored-row conversion notices — resyncs are + * event-driven, so a legacy row would otherwise re-warn on every publish. + */ + private storedConversionWarned = new Set(); + + /** + * Canonicalize a `sys_metadata` body read directly by this plugin (#3903). + * + * The authored-hook/-action re-syncs read the table themselves (they must — + * env-scoped kernels surface these rows nowhere else), which makes them + * stored-metadata rehydration seams: the full ADR-0087 chain replays here, + * exactly like `protocol.loadMetaFromDb` / `getMetaItems`, so the engine's + * runner and dispatch only ever see canonical shapes (e.g. a pre-17 action + * row's `execute` reads as `target`). + */ + private convertStoredRow(ctx: PluginContext, type: string, data: any): any { + return applyConversionsToStoredItem(type, data, { + onNotice: (n) => { + const key = `${n.conversionId}|${type}|${String(data?.name ?? '')}`; + if (this.storedConversionWarned.has(key)) return; + this.storedConversionWarned.add(key); + ctx.logger.warn( + `[ObjectQLPlugin] stored ${type}/${String(data?.name ?? '')} carries a pre-protocol shape; ${n.message}`, + ); + }, + }); + } + /** * Read the ACTIVE runtime-authored hook rows from `sys_metadata`. * @@ -1278,7 +1311,11 @@ export class ObjectQLPlugin implements Plugin { const hooks: any[] = []; for (const row of rows) { try { - const data = typeof row.metadata === 'string' ? JSON.parse(row.metadata) : row.metadata; + const data = this.convertStoredRow( + ctx, + 'hook', + typeof row.metadata === 'string' ? JSON.parse(row.metadata) : row.metadata, + ); if (!data || typeof data !== 'object' || typeof data.name !== 'string') continue; // Surface the persisted package binding (parity with getMetaItems) // so provenance-aware consumers of the bound hook can read it. @@ -1448,9 +1485,13 @@ export class ObjectQLPlugin implements Plugin { */ private async readAuthoredActionRows(ctx: PluginContext): Promise { if (!this.ql) return null; - const parseRow = (row: any): any | undefined => { + const parseRow = (row: any, type: string): any | undefined => { try { - const data = typeof row.metadata === 'string' ? JSON.parse(row.metadata) : row.metadata; + const data = this.convertStoredRow( + ctx, + type, + typeof row.metadata === 'string' ? JSON.parse(row.metadata) : row.metadata, + ); if (!data || typeof data !== 'object') return undefined; const recPkg = row.package_id ?? undefined; if (recPkg && data._packageId === undefined) data._packageId = recPkg; @@ -1470,16 +1511,18 @@ export class ObjectQLPlugin implements Plugin { } const actions: any[] = []; for (const row of rows) { - const data = parseRow(row); + const data = parseRow(row, 'action'); if (data && typeof data.name === 'string') actions.push(data); } // Embedded shape: authored object rows may carry their own actions. + // Converting the OBJECT row canonicalizes the embedded actions too — + // the chain's action walker covers `objects[].actions[]`. const objectRows: any[] = (await this.ql.find('sys_metadata', { where: { type: 'object', state: 'active' }, })) ?? []; for (const row of objectRows) { - const obj = parseRow(row); + const obj = parseRow(row, 'object'); if (!obj || typeof obj.name !== 'string' || !Array.isArray(obj.actions)) continue; for (const action of obj.actions) { if (!action || typeof action !== 'object' || typeof action.name !== 'string') continue; diff --git a/packages/objectql/src/registry.ts b/packages/objectql/src/registry.ts index 7afa563383..e2b4ee4feb 100644 --- a/packages/objectql/src/registry.ts +++ b/packages/objectql/src/registry.ts @@ -1102,11 +1102,25 @@ export class SchemaRegistry { // so both load paths produce identical lock state. applyProtection(item as any, { packageId }); - // Validation Hook + // Spec-conformance DIAGNOSTIC — deliberately not a gate (#3903). + // + // Registration proceeds on failure because refusing here would unhook the + // item from every serving surface: an object row backs live tables, and an + // unregistered item cannot even be listed, opened, or fixed in Studio — + // enforcement at this seam turns bad metadata into a data outage. The + // contract is enforced where an author is present to act: the write path + // (`saveMetaItem` → 422 with structured issues) rejects new off-spec + // bodies, the read path badges rows via `_diagnostics`, and every stored + // row is canonicalized by the ADR-0087 conversion chain BEFORE it reaches + // this method — so what fails here is a genuine, current-contract + // violation, not chain-owned history. try { this.validate(type, item); } catch (e: any) { - console.error(`[Registry] Validation failed for ${type} ${baseName}: ${e.message}`); + console.error( + `[Registry] [metadata_spec_invalid] ${type}/${baseName} fails the current spec schema: ${e.message} — ` + + `registered anyway (serving availability); fix it at the source or via Studio (reads carry _diagnostics).`, + ); } // Use composite key (packageId:name) when packageId is provided diff --git a/packages/objectql/src/validation/rule-validator.test.ts b/packages/objectql/src/validation/rule-validator.test.ts index e882e0e307..102454bcc3 100644 --- a/packages/objectql/src/validation/rule-validator.test.ts +++ b/packages/objectql/src/validation/rule-validator.test.ts @@ -57,9 +57,23 @@ describe('field requiredWhen enforcement (B2)', () => { it('a write that FIXES the legacy violation passes', () => { expect(() => evaluateValidationRules(invoiceFields, { amount: 25 }, 'update', { previous: { status: 'sent' } })).not.toThrow(); }); - it('honors the conditionalRequired alias', () => { - const f = { fields: { amount: { type: 'currency', conditionalRequired: "record.status == 'sent'" } } }; - expect(() => evaluateValidationRules(f, { status: 'sent' }, 'insert')).toThrow(ValidationError); + // #3903 — the `conditionalRequired` alias read is RETIRED. Stored pre-17 + // rows are lowered to `requiredWhen` by the ADR-0087 chain at rehydration + // (`applyConversionsToStoredItem`), so this validator only ever sees the + // canonical key; raw legacy input no longer gets a consumer-side fallback. + it('does NOT read the retired conditionalRequired alias (PD #12 — no dialect fallback)', () => { + const f = { fields: { amount: { type: 'currency', conditionalRequired: "record.status == 'sent'" } } } as any; + expect(() => evaluateValidationRules(f, { status: 'sent' }, 'insert')).not.toThrow(); + }); + it('enforces the rule once the stored-conversion chain has lowered the alias', async () => { + const { applyConversionsToStoredItem } = await import('@objectstack/spec'); + const storedRow = { + name: 'invoice', + fields: { amount: { type: 'currency', conditionalRequired: "record.status == 'sent'" } }, + }; + const converted = applyConversionsToStoredItem('object', storedRow) as any; + expect(converted.fields.amount.requiredWhen).toBe("record.status == 'sent'"); + expect(() => evaluateValidationRules(converted, { status: 'sent' }, 'insert')).toThrow(ValidationError); }); }); diff --git a/packages/objectql/src/validation/rule-validator.ts b/packages/objectql/src/validation/rule-validator.ts index d2333fa7ef..0d8c7dccd2 100644 --- a/packages/objectql/src/validation/rule-validator.ts +++ b/packages/objectql/src/validation/rule-validator.ts @@ -437,19 +437,15 @@ interface ConditionalFieldOption { interface ConditionalFieldDef { /** Author-declared display label, used to name the field in messages (#3957). */ label?: string; + // The ONLY predicate slot. The retired `conditionalRequired` alias (#3754 / + // #3855) is no longer read here: every path that hands this validator a + // STORED field definition now replays the ADR-0087 conversion chain at + // rehydration (#3903 — `applyConversionsToStoredItem`, retired entries + // included), so a pre-17 row arrives with the alias already lowered into + // `requiredWhen`. Authored metadata never carried it (FieldSchema + // tombstones the key). A caller handing raw, unconverted legacy input is + // off-contract — PD #12: no consumer-side dialect fallback returns here. requiredWhen?: string | Expression; - // Retired authorable key. #3754 lowered it into `requiredWhen` and dropped it, - // so PARSED metadata never carried it; #3855 removed it from the spec outright, - // so it can no longer be AUTHORED at all — `FieldSchema` rejects it. - // - // The alias-reading fallback below stays deliberately, and its job has changed: - // this validator is also handed RAW, unparsed field definitions, which includes - // metadata STORED under protocol <= 16. Dropping the read here would silently - // stop enforcing those rules on existing deployments — a runtime regression the - // spec change does not otherwise cause. It retires when that stored metadata is - // migrated (`os migrate meta`), not when the spec key went away. - // Canonical first, always — never `conditionalRequired ?? requiredWhen`. - conditionalRequired?: string | Expression; readonlyWhen?: string | Expression; /** Static, unconditional read-only flag (`field.readonly`). #2948. */ readonly?: boolean; @@ -487,7 +483,7 @@ function isMissing(v: unknown): boolean { function fieldsNeedPrior(fields: Record | undefined): boolean { if (!fields) return false; return Object.values(fields).some( - (f) => f && (f.requiredWhen || f.conditionalRequired || f.readonlyWhen || fieldHasOptionVisibility(f)), + (f) => f && (f.requiredWhen || f.readonlyWhen || fieldHasOptionVisibility(f)), ); } @@ -603,10 +599,10 @@ export function evaluateValidationRules( const errors: FieldValidationError[] = []; // Field-level conditional rules (B2): a field whose `requiredWhen` - // (or its `conditionalRequired` alias) predicate is TRUE over the merged - // record must have a value — enforced server-side so the rule can't be - // bypassed. (`readonlyWhen` is handled by stripReadonlyWhenFields on the - // write path, not here.) A broken predicate is fail-open (logged, skipped). + // predicate is TRUE over the merged record must have a value — enforced + // server-side so the rule can't be bypassed. (`readonlyWhen` is handled by + // stripReadonlyWhenFields on the write path, not here.) A broken predicate + // is fail-open (logged, skipped). // // ADR-0113 non-regression: reject iff the MERGED state violates AND the // PRE state complied. A write may not take the record from compliant to @@ -618,7 +614,7 @@ export function evaluateValidationRules( // conditional contract on a deployed object safe (#3929's objection). if (hasFieldRules && fields) { for (const [name, def] of Object.entries(fields)) { - const pred = def?.requiredWhen ?? def?.conditionalRequired; + const pred = def?.requiredWhen; if (!pred) continue; const res = ExpressionEngine.evaluate(toExpression(pred), { record: merged, previous }); if (!res.ok) { diff --git a/packages/services/service-automation/src/engine.ts b/packages/services/service-automation/src/engine.ts index 3672145b50..c5b676057b 100644 --- a/packages/services/service-automation/src/engine.ts +++ b/packages/services/service-automation/src/engine.ts @@ -1408,6 +1408,16 @@ export class AutomationEngine implements IAutomationService { // live executor registry: an open-namespace node-type rename over a type // a custom executor owns becomes a loud, refused conflict — never a // silent clobber of the third party's node. + // + // `includeRetired` (#3903): this seam rehydrates DATA AT REST — flows + // stored in `sys_metadata` outlive any load window, and a row written + // under protocol N has no author for a tombstone to teach. When a flow + // conversion graduates to `retiredFromLoadPath` (retiring at N+1), the + // stored flows that still carry the old shape must keep canonicalizing + // here or the retirement would silently change their behavior — the + // exact hazard this seam exists to prevent. Authored sources keep + // window semantics at their own seam (`normalizeStackInput` applies + // live-window entries only; the schema tombstones the retired shape). const reservedNodeTypes = new Set([ ...FLOW_STRUCTURAL_NODE_TYPES, ...this.nodeExecutors.keys(), @@ -1415,6 +1425,7 @@ export class AutomationEngine implements IAutomationService { ]); const converted = applyConversionsToFlow(definition, { reservedNodeTypes, + includeRetired: true, onNotice: (n) => this.logger.warn(`[flow '${name}'] ${n.code}: ${n.message}`), onConflict: (c) => this.logger.warn(`[flow '${name}'] ${c.code}: ${c.message}`), }); diff --git a/packages/spec/api-surface.json b/packages/spec/api-surface.json index 41a8e22c07..02f07d4f21 100644 --- a/packages/spec/api-surface.json +++ b/packages/spec/api-surface.json @@ -105,6 +105,7 @@ "SpecSurfaceAddSchema (const)", "SpecSurfaceRemove (type)", "SpecSurfaceRemoveSchema (const)", + "StoredConversionOptions (type)", "SurfaceDiff (interface)", "TemplateExpressionInputSchema (const)", "Tool (type)", @@ -112,6 +113,7 @@ "ViewKeyCollision (interface)", "applyConversions (function)", "applyConversionsToFlow (function)", + "applyConversionsToStoredItem (function)", "applyMetaMigrations (function)", "cel (function)", "classifyRequiredCapability (function)", diff --git a/packages/spec/src/conversions/index.ts b/packages/spec/src/conversions/index.ts index 5af7dcf94f..e2b2f4ffd6 100644 --- a/packages/spec/src/conversions/index.ts +++ b/packages/spec/src/conversions/index.ts @@ -27,3 +27,7 @@ export { collectConversionNotices, type ApplyConversionsOptions, } from './apply.js'; +export { + applyConversionsToStoredItem, + type StoredConversionOptions, +} from './stored.js'; diff --git a/packages/spec/src/conversions/stored.test.ts b/packages/spec/src/conversions/stored.test.ts new file mode 100644 index 0000000000..9c09fcb0ae --- /dev/null +++ b/packages/spec/src/conversions/stored.test.ts @@ -0,0 +1,91 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +import { describe, expect, it } from 'vitest'; + +import { applyConversionsToStoredItem } from './stored.js'; +import type { ConversionNotice } from './types.js'; + +// The stored pass replays the FULL chain (ADR-0087 addendum, #3903): a row at +// rest was written under some past protocol and has no author to be taught by +// a tombstone, so `retiredFromLoadPath` entries — which the authored load seam +// skips — MUST apply here. +describe('applyConversionsToStoredItem (stored sys_metadata rows, #3903)', () => { + it('replays a RETIRED conversion over a stored object row (conditionalRequired → requiredWhen)', () => { + const row = { + name: 'crm_task', + label: 'Task', + fields: { due_date: { type: 'date', conditionalRequired: 'record.stage == "closed"' } }, + }; + const notices: ConversionNotice[] = []; + const out = applyConversionsToStoredItem('object', row, { onNotice: (n) => notices.push(n) }); + expect(out.fields.due_date).toEqual({ type: 'date', requiredWhen: 'record.stage == "closed"' }); + expect(notices.map((n) => n.conversionId)).toContain('field-conditionalRequired-to-requiredWhen'); + }); + + it('replays a retired conversion over a standalone stored action row (execute → target)', () => { + const row = { name: 'convert', label: 'Convert', type: 'script', execute: 'convertHandler' }; + const out = applyConversionsToStoredItem('action', row) as Record; + expect(out.target).toBe('convertHandler'); + expect('execute' in out).toBe(false); + }); + + it('reaches actions EMBEDDED in a stored object row', () => { + const row = { + name: 'crm_lead', + label: 'Lead', + actions: [{ name: 'convert', type: 'script', execute: 'convertHandler' }], + }; + const out = applyConversionsToStoredItem('object', row) as { actions: Array> }; + expect(out.actions[0]!.target).toBe('convertHandler'); + expect('execute' in out.actions[0]!).toBe(false); + }); + + it('accepts the legacy PLURAL row-type spelling', () => { + const row = { name: 'crm_deal', label: 'Deal', sharingModel: 'read' }; + const out = applyConversionsToStoredItem('objects', row) as Record; + expect(out.sharingModel).toBe('public_read'); + }); + + it('is idempotent — a canonical row passes through by reference', () => { + const row = { + name: 'crm_task', + label: 'Task', + fields: { due_date: { type: 'date', requiredWhen: 'record.stage == "closed"' } }, + }; + expect(applyConversionsToStoredItem('object', row)).toBe(row); + }); + + it('never mutates the stored input', () => { + const row = { + name: 'crm_task', + fields: { due_date: { type: 'date', conditionalRequired: 'record.x == 1' } }, + }; + const snapshot = structuredClone(row); + applyConversionsToStoredItem('object', row); + expect(row).toEqual(snapshot); + }); + + it('passes through unknown types and non-object items unchanged', () => { + const row = { name: 'h', object: 'crm_lead', event: 'afterInsert' }; + expect(applyConversionsToStoredItem('some_plugin_type', row)).toBe(row); + expect(applyConversionsToStoredItem('object', null)).toBe(null); + expect(applyConversionsToStoredItem('object', 'not-a-dict')).toBe('not-a-dict'); + const arr = [{ name: 'x' }]; + expect(applyConversionsToStoredItem('object', arr)).toBe(arr); + }); + + it('threads the conflict guard context through (flow callers that own a registry)', () => { + const flow = { + name: 'notify_flow', + nodes: [{ id: 'n1', type: 'webhook', config: { url: 'https://hooks.example.com' } }], + }; + const conflicts: string[] = []; + const out = applyConversionsToStoredItem('flow', flow, { + reservedNodeTypes: new Set(['webhook']), + onConflict: (c) => conflicts.push(c.token), + }) as typeof flow; + // A live executor owns 'webhook' → the rename is refused and reported. + expect(out.nodes[0]!.type).toBe('webhook'); + expect(conflicts).toEqual(['webhook']); + }); +}); diff --git a/packages/spec/src/conversions/stored.ts b/packages/spec/src/conversions/stored.ts new file mode 100644 index 0000000000..d2a13255e0 --- /dev/null +++ b/packages/spec/src/conversions/stored.ts @@ -0,0 +1,79 @@ +// Copyright (c) 2025 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The stored-metadata conversion pass (ADR-0087 D2 addendum, #3903). + * + * Every other entry point into the conversion layer serves **authored + * source** — `normalizeStackInput` for `defineStack`/`validate`/`lint`, the + * migration chain for `migrate meta`. Metadata **at rest** (`sys_metadata` + * rows written by Studio or the runtime authoring APIs) is a different + * compatibility problem: a row written under protocol N sits unchanged while + * the platform moves to N+1, N+2, … and there is no author in the loop to + * read a tombstone or run the chain against it. + * + * {@link applyConversionsToStoredItem} is the one primitive every stored-row + * rehydration seam calls, and it encodes the policy for data at rest: + * + * - **The full chain replays, including `retiredFromLoadPath` entries.** + * Retirement is an *authoring-surface* event — the schema tombstones the + * key so a live author is taught the canonical spelling. A stored row has + * no author; refusing its historical shape would not teach anyone, it + * would only break data that once worked. D3 keeps every conversion + * forever precisely so any past major can replay forward — a row at rest + * is the perpetual "consumer arriving late". + * - **Idempotent by construction.** Conversions only rewrite shapes they + * positively recognize, so re-converting an already-canonical item is a + * no-op — seams may safely stack (e.g. a row converted here and again at + * `AutomationEngine.registerFlow`). + * - **Never throws, never validates.** Like {@link applyConversions}, this + * only rewrites; gating stays where it belongs (the write path's schema + * gate, and the caller's own parse where execution demands one). + * + * The **write** path must NOT use this: `saveMetaItem` rejects off-spec + * bodies with the current schema (tombstones included), which is what keeps + * new rows canonical and makes this pass a strictly shrinking concern. + */ + +import { applyConversions, type ApplyConversionsOptions } from './apply.js'; +import { PLURAL_TO_SINGULAR, SINGULAR_TO_PLURAL } from '../shared/metadata-collection.zod.js'; + +/** + * Options for {@link applyConversionsToStoredItem} — everything + * {@link ApplyConversionsOptions} offers except `includeRetired`, which this + * seam pins to `true` (the whole point of the stored pass; see module doc). + */ +export type StoredConversionOptions = Omit; + +/** + * Canonicalize a single stored metadata item of the given type by replaying + * the **full** ADR-0087 conversion chain (retired entries included) over it. + * + * `type` accepts the singular metadata type name (`'object'`, `'view'`, + * `'action'`, …) or its legacy plural row spelling (`'objects'`) — both map + * to the stack collection the conversion walkers target. A type with no + * stack collection (e.g. `'hook'`-adjacent plugin types) and a non-object + * item pass through unchanged: this seam never manufactures shape. + * + * Deliberately NOT applied to `'flow'` by the metadata-protocol seams: + * flow-node conversions carry an open-namespace conflict guard that needs + * the live executor registry (`reservedNodeTypes`), which only the + * automation engine has — flows canonicalize at `registerFlow` instead, + * with the same full-chain policy. Callers that *do* have the context may + * pass it via `options.reservedNodeTypes` and convert flows here. + */ +export function applyConversionsToStoredItem( + type: string, + item: T, + options: StoredConversionOptions = {}, +): T { + if (item == null || typeof item !== 'object' || Array.isArray(item)) return item; + const singular = PLURAL_TO_SINGULAR[type] ?? type; + const collection = SINGULAR_TO_PLURAL[singular]; + if (!collection) return item; + const converted = applyConversions( + { [collection]: [item as Record] }, + { ...options, includeRetired: true }, + ); + const items = converted[collection]; + return Array.isArray(items) && items.length > 0 ? (items[0] as T) : item; +}