diff --git a/.changeset/duplicate-package-flow-canonicalization.md b/.changeset/duplicate-package-flow-canonicalization.md new file mode 100644 index 0000000000..486ef32708 --- /dev/null +++ b/.changeset/duplicate-package-flow-canonicalization.md @@ -0,0 +1,61 @@ +--- +"@objectstack/metadata-protocol": patch +"@objectstack/cli": patch +--- + +fix(metadata-protocol): `duplicatePackage` stops minting pre-protocol flow rows (#4498) + +`duplicatePackage` canonicalizes each source row before re-saving it, under a +stated guarantee: "duplication never mints new rows in a pre-protocol dialect." +It delivered that through `convertStoredItem`, which opens with +`if (singular === 'flow') return { item: data, notices: [] }` — so for flows the +guarantee was **not** delivered. + +It did not fail loudly either. `FlowNodeSchema.config` is an open `z.record`, so +a pre-17 body (a `delete_record` carrying `config.filters`) sails through +`saveMetaItem`'s schema gate and lands verbatim in a brand-new row. + +**Why this mattered more than an un-migrated row.** ADR-0087 justifies the whole +stored-metadata design on new writes always being canonical, *therefore* the +stored pass being "a strictly shrinking concern". `duplicatePackage` was a live +producer contradicting that for flows: an operator could run +`os migrate meta --stored --apply`, get a clean report, duplicate a package, and +be back to having pre-protocol rows — with the report still saying protocol N +until the next run. + +**The capability was already reachable.** The reason for the flow skip is real — +flow-node conversions carry ADR-0078's open-namespace conflict guard, which needs +the automation engine's live executor registry to tell a rename from a clobber. +But the protocol is constructed with an accessor for the kernel's service table +(the same one `analytics` and `package` are read from), and the automation +service registers under `automation`. A new private `resolveFlowCanonicalizer` +reads `canonicalizeStoredFlow` (#4454) off it, so every caller running next to a +live engine gets flow coverage without threading anything. + +- **`duplicatePackage`** canonicalizes flow rows through it. A refused rename + fails that item into the existing `failed[]` naming the token — copying the + un-renamed body would mint exactly the row this fixes. A flow that cannot + canonicalize fails the same way. With no engine reachable (a control-plane or + metadata-only host) the source body is copied as-is: no worse than the source + row already is, and failing an unrelated duplication over it would be its own + regression. +- **`migrateStoredMetadata`'s `canonicalizeFlow` becomes an override.** It now + defaults to the resolver. The CLI stopped passing one — it boots its inert + engine into the same kernel, so both routes reached the same instance, and two + routes to one capability is how they drift. The parameter stays for callers + with no registry and for testing the flow branch without an engine. +- **Resolution is lazy, per call.** Plugin init order does not guarantee + `automation` is in the table when the protocol is assembled (the CLI adds it + after ObjectQL by design), so caching `undefined` from a too-early read would + disable flow canonicalization for the life of the process. + +Two smaller honesty fixes ride along: a source item that fails *conversion* (a +tombstoned key throws) is now reported as such instead of as `unparseable +metadata`, and `migrateStoredMetadata`'s "no engine" skip reason says no +automation service is reachable rather than blaming the caller for not supplying +one. + +Reads are unchanged. `getMetaItems` / `getMetaItem` / `getMetaItemLayered` / +`loadMetaFromDb` still skip flows — they are reads, covered by `registerFlow` +canonicalizing at execution, and are not producing bad data. Duplication was the +one that writes. diff --git a/.changeset/meta-migrate-stored-route.md b/.changeset/meta-migrate-stored-route.md new file mode 100644 index 0000000000..f46edb7492 --- /dev/null +++ b/.changeset/meta-migrate-stored-route.md @@ -0,0 +1,47 @@ +--- +"@objectstack/rest": minor +"@objectstack/runtime": minor +"@objectstack/client": minor +--- + +feat(rest,runtime,client): `POST /meta/_migrate-stored` — run the stored-metadata migration without a shell (#4327) + +`os migrate meta --stored` (#4327) gave ADR-0087's stored-metadata chain a finish +line, but only for someone who can reach the deployment's database from a +terminal. A hosted operator cannot, so on a managed deployment the chain had no +finish line at all — just the per-read conversion, running forever, with no way +to assert what protocol the rows are on. + +The same pass is now reachable over HTTP: + +```ts +const preview = await client.meta.migrateStored(); // writes nothing +const result = await client.meta.migrateStored({ apply: true }); +const flows = await client.meta.migrateStored({ types: ['flow'] }); +``` + +It returns the same `StoredMigrationReport` the CLI renders, and takes the same +posture: + +- **Preview by default.** `apply` must be literally `true`; an empty body, a + missing body, and `"apply": "yes"` all preview. Nothing is inferred. +- **Gated on `manage_metadata`.** Unlike the single-item `PUT /meta/:type/:name` + next door, this rewrites every eligible row in the deployment, so it demands + the ADR-0066 D1 authoring capability rather than just a session, and answers + `403` otherwise. The gate runs before the protocol is probed, so an + unauthorized caller cannot use `403`-vs-`501` to learn which kernels can be + migrated. `/meta`'s anonymous-deny umbrella still closes it to anonymous + callers first. +- **Attributed to the caller.** The `actor` recorded on the history and audit + rows names the user who fired it — that is the question those rows exist to + answer. + +**Flows need no extra setup on this path.** The CLI has to boot an inert +automation engine to hold the executor registry ADR-0078's conflict guard needs; +a server already has a live one, and the protocol resolves it from the services +registry itself (#4498), so this route covers flow rows by simply running in the +process that owns them. + +Registered on both the REST server and the runtime dispatcher's `/meta` domain, +ledgered in both route ledgers, and mounted before `/:type` so the +leading-underscore segment is never captured as a metadata type name. diff --git a/content/docs/api/client-sdk.mdx b/content/docs/api/client-sdk.mdx index bb473d6494..188d3656a4 100644 --- a/content/docs/api/client-sdk.mdx +++ b/content/docs/api/client-sdk.mdx @@ -187,6 +187,10 @@ const bad = await client.meta.getDiagnostics({ severity: 'error' }); const refs = await client.meta.getReferences('object', 'account'); const trail = await client.meta.getAudit('object', 'account', { limit: 20 }); const tree = await client.meta.getBookTree('handbook'); + +// Operator: rewrite stored rows into today's canonical shape (ADR-0087). +// Preview unless `apply: true`; requires the `manage_metadata` capability. +const report = await client.meta.migrateStored({ apply: true }); ``` ### `client.data` — CRUD & Batch diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index f1420526ae..eaf0a8f956 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -846,6 +846,31 @@ rewrites an **author's source** and reads no database; `--stored` rewrites **one deployment's rows** and reads no config. Same chain, opposite ends of the contract — which is why the two modes are mutually exclusive. +**Without shell access, use the route.** This command needs to reach the +deployment's database directly, which a hosted operator cannot do. The same pass +is exposed over HTTP: + +```http +POST /api/v1/meta/_migrate-stored +Content-Type: application/json + +{ "apply": true, "types": ["flow"] } +``` + +or from the SDK: + +```ts +const preview = await client.meta.migrateStored(); // writes nothing +const result = await client.meta.migrateStored({ apply: true }); +``` + +It returns the same report the CLI renders, and takes the same posture: +**preview unless `apply` is literally `true`**, `types` optional. It requires the +`manage_metadata` capability — it rewrites every eligible row in the deployment, +not one item — and answers `403` otherwise. Flows need no extra setup on this +path: the server already holds a live automation engine, so the run resolves the +executor registry the conflict guard needs from the process it is running in. + ### Scaffolding | Command | Alias | Description | diff --git a/docs/adr/0087-metadata-protocol-upgrade-contract.md b/docs/adr/0087-metadata-protocol-upgrade-contract.md index 58764b6dcc..ad1bbf46ab 100644 --- a/docs/adr/0087-metadata-protocol-upgrade-contract.md +++ b/docs/adr/0087-metadata-protocol-upgrade-contract.md @@ -487,3 +487,48 @@ Closing it needed three decisions. A refused rename — the guard firing because the old token is a live name owned by something else — fails that row loudly with the token and its owner. Never a silent skip, never a clobber; that is the whole reason the guard exists. + +## Addendum (2026-08-01c) — "strictly shrinking" was false for flows (#4498) + +The bullet above claims new rows are always canonical, *therefore* the stored +pass is a strictly shrinking concern. `duplicatePackage` was a live producer +contradicting it: it canonicalizes each source row before re-saving, but through +`convertStoredItem`, which returns `flow` bodies untouched. `FlowNodeSchema.config` +is an open `z.record`, so a pre-17 body sailed through `saveMetaItem`'s gate and +landed verbatim in a brand-new row. An operator could run the migration, get a +clean report, duplicate a package, and be back to pre-protocol rows — with the +report still saying protocol N until the next run. + +- **The capability was already reachable; only the wiring was missing.** The + protocol is constructed with an accessor for the kernel's service table (the + same one `analytics` and `package` are read from), and the automation service + registers under `automation`. `resolveFlowCanonicalizer` reads + `canonicalizeStoredFlow` off it. So the fix is not new plumbing per call site + — it is one private resolver that every caller running next to a live engine + shares. +- **The explicit hook becomes an override, not a requirement.** + `migrateStoredMetadata`'s `canonicalizeFlow` defaults to the resolver, so the + CLI stopped passing one (it boots the inert engine into the same kernel, so + both routes reached the same instance — two routes to one capability is how + they drift). The parameter stays for callers with no registry and for testing + the flow branch without an engine. +- **Resolution is lazy, per call.** Plugin init order does not guarantee + `automation` is in the table when the protocol is assembled — the CLI adds it + after ObjectQL by design — so caching `undefined` from a too-early read would + disable flow canonicalization for the life of the process. +- **The failure posture matches #4454's.** A refused rename fails that item into + `duplicatePackage`'s existing `failed[]` naming the token, rather than copying + the un-renamed body: producing exactly the row this fix exists to prevent is + the one outcome worse than failing the copy. A flow that cannot canonicalize + at all fails the same way. With **no** engine reachable (a control-plane or + metadata-only host) the source body is copied as-is — no worse than the source + row already is, and failing an unrelated duplication over it would be its own + regression. +- **Reads were not changed.** `getMetaItems` / `getMetaItem` / + `getMetaItemLayered` / `loadMetaFromDb` still skip flows; they are reads, + covered by `registerFlow` canonicalizing at execution, and are not producing + bad data. Duplication was the one that *writes*. The resolver is the seam they + would adopt if that changes. + +The premise is restored rather than restated: the stored pass shrinks because +every write path now canonicalizes, not because the sentence says so. diff --git a/packages/cli/src/commands/migrate/meta.stored-flow-resolution.integration.test.ts b/packages/cli/src/commands/migrate/meta.stored-flow-resolution.integration.test.ts new file mode 100644 index 0000000000..249b10568c --- /dev/null +++ b/packages/cli/src/commands/migrate/meta.stored-flow-resolution.integration.test.ts @@ -0,0 +1,185 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * End-to-end acceptance for #4498's CLI half: `os migrate meta --stored` still + * covers `flow` rows after the command stopped threading `canonicalizeFlow`. + * + * #4454 wired flow coverage by resolving `automation` off the booted kernel in + * the command body and handing `canonicalizeStoredFlow` to + * `migrateStoredMetadata`. #4498 gave the protocol its own resolver — it is + * constructed with an accessor for the kernel's service table, which is the + * same table the inert engine registers into — so the command passes nothing + * and the redundant second route is gone. + * + * That is exactly the kind of removal a unit test cannot defend: every flag test + * still passes if the protocol silently fails to find the engine, and the only + * symptom is flow rows quietly reporting `skipped` again. So this boots the + * REAL stack the command boots (`bootSchemaStack` + + * `buildDataMigrationPlugins({ automation: true })`), seeds a pre-17 flow row, + * and asserts the rewrite lands in the database. + */ + +import { describe, it, expect, beforeEach, afterEach } from 'vitest'; +import { mkdtempSync, writeFileSync, mkdirSync, rmSync } from 'node:fs'; +import { tmpdir } from 'node:os'; +import { join } from 'node:path'; +import type { IObjectQLEngine } from '@objectstack/spec/contracts'; +import { bootSchemaStack } from '../../utils/schema-migrate.js'; +import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js'; + +/** + * `SchemaStack.kernel` is untyped, so a type argument is a TS2347 — the slot's + * contract is stated on the RESULT instead. Narrowing, not erasing: `: any` + * here would switch off checking on every `ql.*` call below while looking + * identical to code that has it (the `slot-lookup` rule's whole point). + */ +const engineOf = (stack: { kernel: any }): IObjectQLEngine => + stack.kernel.getService('objectql') as IObjectQLEngine; + +/** Elevated so the seed write bypasses RLS on a system object. */ +const SYSTEM = { context: { isSystem: true } }; + +const ARTIFACT = { + id: 'stored_flow_smoke', + name: 'Stored Flow Smoke', + objects: [{ name: 'sfs_lead', fields: { title: { type: 'text' } } }], +}; + +/** + * A pre-17 flow: `delete_record` carrying `config.filters`, which the + * `flow-node-crud-filter-alias` conversion (toMajor 11) renames to `filter`. + * Written straight into `sys_metadata`, bypassing today's schema gate — exactly + * like a row saved years ago under an older protocol. + */ +const LEGACY_FLOW = { + name: 'sfs_purge', + label: 'Purge Stale Leads', + type: 'autolaunched', + status: 'active', + nodes: [ + { id: 'n0', type: 'start', label: 'Start' }, + { + id: 'n1', + type: 'delete_record', + label: 'Purge', + config: { objectName: 'sfs_lead', filters: { title: 'stale' } }, + }, + ], + edges: [{ id: 'e1', source: 'n0', target: 'n1' }], +}; + +describe('os migrate meta --stored — the protocol resolves the engine itself (#4498)', () => { + let dir: string; + let dbFile: string; + const savedEnv: Record = {}; + + beforeEach(() => { + dir = mkdtempSync(join(tmpdir(), 'os-stored-flow-')); + mkdirSync(join(dir, 'dist'), { recursive: true }); + mkdirSync(join(dir, 'data'), { recursive: true }); + dbFile = join(dir, 'data', 'app.db'); + writeFileSync(join(dir, 'dist', 'objectstack.json'), JSON.stringify(ARTIFACT)); + + savedEnv.OS_ARTIFACT_PATH = process.env.OS_ARTIFACT_PATH; + savedEnv.NODE_ENV = process.env.NODE_ENV; + process.env.OS_ARTIFACT_PATH = join(dir, 'dist', 'objectstack.json'); + process.env.NODE_ENV = 'production'; + }); + + afterEach(() => { + process.env.OS_ARTIFACT_PATH = savedEnv.OS_ARTIFACT_PATH; + process.env.NODE_ENV = savedEnv.NODE_ENV; + try { rmSync(dir, { recursive: true, force: true }); } catch { /* ignore */ } + }); + + it('rewrites a pre-17 flow row with NO canonicalizeFlow passed by the command', async () => { + const stack = await bootSchemaStack({ + databaseUrl: `file:${dbFile}`, + projectRoot: dir, + extraPlugins: await buildDataMigrationPlugins({ automation: true }), + }); + try { + const ql = engineOf(stack); + await ql.insert('sys_metadata', { + type: 'flow', + name: 'sfs_purge', + state: 'active', + metadata: JSON.stringify(LEGACY_FLOW), + }, SYSTEM); + + const protocol: any = stack.kernel.getService('protocol'); + + // The command's exact call since #4498 — no `canonicalizeFlow`. + const report = await protocol.migrateStoredMetadata({ + apply: true, + types: ['flow'], + actor: 'os migrate meta --stored', + }); + + // Before the resolver this row came back `skipped` with "no automation + // service is reachable". Asserted as the REASON rather than as a bare + // count, so a regression here says what went wrong instead of just + // "expected 1 to be 0". + expect( + report.rows + .filter((r: any) => r.outcome === 'skipped' || r.outcome === 'failed') + .map((r: any) => `${r.outcome}: ${r.reason}`), + ).toEqual([]); + expect(report.skipped).toBe(0); + expect(report.failed).toBe(0); + expect(report.rewritten).toBe(1); + + // …and the bytes on disk actually moved. + const [row] = await ql.find('sys_metadata', { + where: { type: 'flow', name: 'sfs_purge', state: 'active' }, + }, SYSTEM); + const stored = typeof row.metadata === 'string' ? JSON.parse(row.metadata) : row.metadata; + const node = stored.nodes.find((n: any) => n.id === 'n1'); + expect(node.config).toEqual({ objectName: 'sfs_lead', filter: { title: 'stale' } }); + expect(node.config).not.toHaveProperty('filters'); + + // The write-back must NOT carry the schema's defaults (#4454): persisting + // a `version` / `runAs` the author never wrote would pin this row to + // today's value while untouched rows follow tomorrow's. + expect(stored).not.toHaveProperty('runAs'); + expect(stored.edges[0]).not.toHaveProperty('isDefault'); + + // A second pass has nothing left to do — the finish line the whole + // feature exists to provide. + const rerun = await protocol.migrateStoredMetadata({ types: ['flow'] }); + expect(rerun.scanned).toBe(1); + expect(rerun.canonical).toBe(1); + expect(rerun.pending).toBe(0); + } finally { + await stack.shutdown(); + } + }, 120_000); + + it('without the automation plugin the row is skipped with the reason, never counted done', async () => { + // The honest negative: the coverage comes from the engine being present, + // not from the report defaulting to optimistic. + const stack = await bootSchemaStack({ + databaseUrl: `file:${dbFile}`, + projectRoot: dir, + extraPlugins: await buildDataMigrationPlugins(), + }); + try { + const ql = engineOf(stack); + await ql.insert('sys_metadata', { + type: 'flow', + name: 'sfs_purge', + state: 'active', + metadata: JSON.stringify(LEGACY_FLOW), + }, SYSTEM); + + const protocol: any = stack.kernel.getService('protocol'); + const report = await protocol.migrateStoredMetadata({ apply: true, types: ['flow'] }); + + expect(report.rewritten).toBe(0); + expect(report.skipped).toBe(1); + expect(report.rows[0].reason).toMatch(/no automation service is reachable/); + } finally { + await stack.shutdown(); + } + }, 120_000); +}); diff --git a/packages/cli/src/commands/migrate/meta.ts b/packages/cli/src/commands/migrate/meta.ts index 90c442291b..d5b9a8dae8 100644 --- a/packages/cli/src/commands/migrate/meta.ts +++ b/packages/cli/src/commands/migrate/meta.ts @@ -30,7 +30,6 @@ import { bootSchemaStack } from '../../utils/schema-migrate.js'; import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js'; import { OCCUPANCY_HINT, probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js'; import { describeOccupancy } from '../../utils/sqlite-occupancy.js'; -import type { IAutomationService } from '@objectstack/spec/contracts'; async function confirm(question: string): Promise { if (!process.stdin.isTTY) return false; // non-interactive → require --yes @@ -545,23 +544,18 @@ export default class MigrateMeta extends Command { const { formatStoredMigrationReport, storedMigrationClean } = await import('@objectstack/metadata-protocol'); - // The automation engine canonicalizes `flow` rows — it holds the executor - // registry ADR-0078's conflict guard needs (#4454). It is booted inert, so - // this is the only thing it does in this process. Absent (an older stack, - // or a boot that skipped it), flow rows keep reporting `skipped` with the - // reason rather than being silently counted done. - // `SchemaStack.kernel` is untyped, so the slot's contract is stated on the - // result rather than as a type argument — narrowing, not erasing. - let automation: IAutomationService | undefined; - try { automation = stack.kernel.getService('automation') as IAutomationService | undefined; } - catch { /* not registered — flows stay reported as skipped */ } - const canonicalize = automation?.canonicalizeStoredFlow?.bind(automation); - + // No `canonicalizeFlow` is threaded from here. The automation engine + // canonicalizes `flow` rows — it holds the executor registry ADR-0078's + // conflict guard needs (#4454) — and the protocol resolves it from the + // kernel's service table itself (#4498), which is the same table the + // inert engine this command boots registers into. Passing it again would + // be a second route to one capability, and the two would drift. + // Absent (an older stack, or a boot that skipped it), flow rows keep + // reporting `skipped` with the reason rather than being counted done. const report = await protocol.migrateStoredMetadata({ apply, ...(flags.type && flags.type.length > 0 ? { types: flags.type } : {}), actor: 'os migrate meta --stored', - ...(canonicalize ? { canonicalizeFlow: canonicalize } : {}), }); const clean = storedMigrationClean(report); if (!clean) exitCode = 1; diff --git a/packages/client/src/index.ts b/packages/client/src/index.ts index dcb3a9a28f..579a8ed4e3 100644 --- a/packages/client/src/index.ts +++ b/packages/client/src/index.ts @@ -653,6 +653,32 @@ export class ObjectStackClient { return this.unwrapResponse(res); }, + /** + * ADR-0087: rewrite stored `sys_metadata` rows into today's canonical + * shape — the server-side form of `os migrate meta --stored` (#4327), + * for operators who cannot reach the deployment's database from a shell. + * + * **Preview by default.** Without `apply: true` this reports what it + * would do and writes nothing; the report is the same + * `StoredMigrationReport` the CLI renders (`scanned` / `canonical` / + * `pending` / `rewritten` / `skipped` / `failed`, plus a `rows` list of + * everything that is not already canonical). + * + * Requires the `manage_metadata` capability (403 otherwise) — it rewrites + * every eligible row in the deployment, not one item. + */ + migrateStored: async (opts?: { apply?: boolean; types?: string[] }) => { + const route = this.getRoute('metadata'); + const res = await this.fetch(`${this.baseUrl}${route}/_migrate-stored`, { + method: 'POST', + body: JSON.stringify({ + ...(opts?.apply === true ? { apply: true } : {}), + ...(opts?.types && opts.types.length > 0 ? { types: opts.types } : {}), + }), + }); + return this.unwrapResponse(res); + }, + /** * ADR-0020 D3.3 FSM introspection: the legal next states for `field` * from state `from`, per the object's `state_machine` validation rule. diff --git a/packages/metadata-protocol/src/protocol.flow-canonicalizer.test.ts b/packages/metadata-protocol/src/protocol.flow-canonicalizer.test.ts new file mode 100644 index 0000000000..1f4e9c9800 --- /dev/null +++ b/packages/metadata-protocol/src/protocol.flow-canonicalizer.test.ts @@ -0,0 +1,327 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #4498 — the flow-skip becomes ONE seam, and `duplicatePackage` stops minting + * pre-protocol flow rows. + * + * `convertStoredItem` returns `flow` bodies untouched, because flow-node + * conversions carry ADR-0078's open-namespace conflict guard and that needs the + * automation engine's live executor registry. #4454 built the capability + * (`AutomationEngine.canonicalizeStoredFlow`) and handed it to + * `migrateStoredMetadata` as an explicit hook, because the CLI has to boot an + * engine of its own to hold one. + * + * Inside a server there is nothing to thread: the protocol is constructed with + * an accessor for the kernel's service table. `resolveFlowCanonicalizer` reads + * the engine from it, which is what these tests pin — and it is what makes the + * skip fixable at `duplicatePackage`, a WRITE that was contradicting ADR-0087's + * "new rows are always canonical, so the stored pass is a strictly shrinking + * concern" on every duplication of a package containing a pre-17 flow. + */ +import { describe, expect, it, vi } from 'vitest'; +import { ObjectStackProtocolImplementation } from './protocol.js'; + +/** A flow body that passes `saveMetaItem`'s schema gate. */ +const flowBody = (config: Record) => ({ + name: 'purge_flow', + label: 'Purge Stale Leads', + type: 'autolaunched', + status: 'active', + nodes: [{ id: 'n1', type: 'delete_record', label: 'Purge', config }], + edges: [], +}); + +/** + * Stands in for `AutomationEngine.canonicalizeStoredFlow` — same contract, + * including the copy-on-write identity an unchanged body comes back with. + */ +const canonicalizeStoredFlow = (_name: string, body: any) => { + const node = body?.nodes?.[0]; + if (!node || !('filters' in (node.config ?? {}))) { + return { storable: body, notices: [], conflicts: [] }; + } + const { filters, ...rest } = node.config; + return { + storable: { ...body, nodes: [{ ...node, config: { ...rest, filter: filters } }] }, + notices: [{ + conversionId: 'flow-node-crud-filter-alias', + surface: 'flow.node.config.filter', + from: 'filters', + to: 'filter', + path: 'flows[0].nodes[0].config', + message: 'filters → filter', + }], + conflicts: [], + }; +}; + +function matches(r: Record, where: Record): boolean { + for (const [k, v] of Object.entries(where)) { + if (v === undefined) continue; + if ((r[k] ?? null) !== v) return false; + } + return true; +} + +/** + * Protocol over a stub engine, with a services table under the caller's + * control — the whole point is that the protocol reads `automation` out of it + * rather than being handed a hook. + */ +function makeProtocol( + rows: Array>, + services: Map = new Map(), +) { + const seeded = rows.map((r, i) => ({ + id: `r_${i + 1}`, + organization_id: null, + package_id: null, + state: 'active', + checksum: `sha256:seed_${i + 1}`, + ...r, + metadata: typeof r.metadata === 'string' ? r.metadata : JSON.stringify(r.metadata), + })); + const engine: any = { + find: vi.fn(async (_t: string, opts?: { where?: Record }) => + seeded.filter((r) => matches(r, opts?.where ?? {}))), + registry: { + getPackage: vi.fn(() => ({ + manifest: { id: 'app.iojn', name: 'Repair', namespace: 'iojn', version: '1.0.0', type: 'application' }, + })), + installPackage: vi.fn(), + }, + }; + const protocol = new ObjectStackProtocolImplementation(engine as never, () => services); + const saveMetaItem = vi.spyOn(protocol, 'saveMetaItem' as never); + (saveMetaItem as any).mockResolvedValue({ success: true } as never); + return { protocol, saveMetaItem, services }; +} + +describe('migrateStoredMetadata resolves the engine itself (#4498)', () => { + const legacyFlowRow = { + type: 'flow', + name: 'purge_flow', + metadata: flowBody({ objectName: 'lead', filters: { status: 'stale' } }), + }; + + it('canonicalizes a flow row with NO canonicalizeFlow threaded by the caller', async () => { + const { protocol, saveMetaItem } = makeProtocol( + [legacyFlowRow], + new Map([['automation', { canonicalizeStoredFlow }]]), + ); + + // No hook. This is the whole claim: a caller running next to a live + // engine — an admin route, a server task — gets flow coverage for free. + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.rewritten).toBe(1); + expect(report.skipped).toBe(0); + const written = (saveMetaItem as any).mock.calls[0][0]; + expect(written.item.nodes[0].config).toEqual({ objectName: 'lead', filter: { status: 'stale' } }); + expect(written.item.nodes[0].config).not.toHaveProperty('filters'); + }); + + it('an already-canonical flow row is counted canonical, not rewritten', async () => { + const { protocol, saveMetaItem } = makeProtocol( + [{ type: 'flow', name: 'purge_flow', metadata: flowBody({ objectName: 'lead', filter: { status: 'stale' } }) }], + new Map([['automation', { canonicalizeStoredFlow }]]), + ); + const report = await protocol.migrateStoredMetadata({ apply: true }); + expect(report.canonical).toBe(1); + expect(report.rewritten).toBe(0); + expect(saveMetaItem).not.toHaveBeenCalled(); + }); + + it('no automation service → skipped with the reason, never counted done', async () => { + const { protocol } = makeProtocol([legacyFlowRow], new Map()); + const report = await protocol.migrateStoredMetadata({ apply: true }); + expect(report.skipped).toBe(1); + expect(report.rewritten).toBe(0); + expect(report.rows[0]!.reason).toMatch(/no automation service is reachable/); + }); + + it('an automation service without canonicalizeStoredFlow is treated as absent', async () => { + // An older service in the slot answers the lookup but not the question. + // Reading that as "flow handled" would report a clean run over rows + // nothing examined. + const { protocol } = makeProtocol([legacyFlowRow], new Map([['automation', { registerFlow() {} }]])); + const report = await protocol.migrateStoredMetadata({ apply: true }); + expect(report.skipped).toBe(1); + expect(report.rows[0]!.reason).toMatch(/no automation service is reachable/); + }); + + it('an explicit canonicalizeFlow overrides the registry one', async () => { + const explicit = vi.fn((_n: string, body: any) => ({ + storable: { ...body, label: 'From the explicit hook' }, + notices: [], + conflicts: [], + })); + const registryHook = vi.fn(canonicalizeStoredFlow); + const { protocol, saveMetaItem } = makeProtocol( + [legacyFlowRow], + new Map([['automation', { canonicalizeStoredFlow: registryHook }]]), + ); + + await protocol.migrateStoredMetadata({ apply: true, canonicalizeFlow: explicit }); + + expect(explicit).toHaveBeenCalledTimes(1); + expect(registryHook).not.toHaveBeenCalled(); + expect((saveMetaItem as any).mock.calls[0][0].item.label).toBe('From the explicit hook'); + }); + + it('resolution is LAZY — a service registered after construction is still found', async () => { + // Plugin init order does not guarantee `automation` is in the table when + // the protocol is assembled (the CLI adds it after ObjectQL by design). + // Caching `undefined` from a too-early read would disable flow + // canonicalization for the life of the process. + const services = new Map(); + const { protocol } = makeProtocol([legacyFlowRow], services); + services.set('automation', { canonicalizeStoredFlow }); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + expect(report.rewritten).toBe(1); + }); + + it('the engine is called with the row NAME, not the body', async () => { + const spy = vi.fn(canonicalizeStoredFlow); + const { protocol } = makeProtocol( + [legacyFlowRow], + new Map([['automation', { canonicalizeStoredFlow: spy }]]), + ); + await protocol.migrateStoredMetadata({ apply: true }); + expect(spy.mock.calls[0][0]).toBe('purge_flow'); + }); +}); + +describe('duplicatePackage canonicalizes flow rows (#4498)', () => { + /** The row from the issue: a pre-17 `delete_record` carrying `config.filters`. */ + const legacyFlowRow = { + type: 'flow', + name: 'iojn_purge_flow', + package_id: 'app.iojn', + metadata: flowBody({ objectName: 'iojn_repair_ticket', filters: { status: 'stale' } }), + }; + const objectRow = { + type: 'object', + name: 'iojn_repair_ticket', + package_id: 'app.iojn', + metadata: { name: 'iojn_repair_ticket', label: 'Ticket', fields: { title: { type: 'text' } } }, + }; + const duplicate = (protocol: any) => protocol.duplicatePackage({ + sourcePackageId: 'app.iojn', + targetPackageId: 'app.iojn2', + targetNamespace: 'iojn2', + }); + + it('the copy lands CANONICAL — the guarantee the comment always claimed', async () => { + const { protocol, saveMetaItem } = makeProtocol( + [legacyFlowRow], + new Map([['automation', { canonicalizeStoredFlow }]]), + ); + + const res = await duplicate(protocol); + + expect(res).toMatchObject({ success: true, copiedCount: 1, failedCount: 0 }); + const written = (saveMetaItem as any).mock.calls[0][0]; + // Before #4498 this was `{ objectName, filters }` verbatim: a brand-new + // row in a pre-protocol dialect, minted by the platform itself. + // (`objectName` is untouched here because this package contains no + // `object` row to rename — the reference rewrite is covered next.) + expect(written.item.nodes[0].config).toEqual({ + objectName: 'iojn_repair_ticket', + filter: { status: 'stale' }, + }); + expect(written.item.nodes[0].config).not.toHaveProperty('filters'); + }); + + it('the reference rewrite still runs ON TOP of the canonical body', async () => { + const { protocol, saveMetaItem } = makeProtocol( + [legacyFlowRow, objectRow], + new Map([['automation', { canonicalizeStoredFlow }]]), + ); + await duplicate(protocol); + const flow = (saveMetaItem as any).mock.calls + .map((c: any) => c[0]) + .find((c: any) => c.type === 'flow'); + // Canonicalized (`filter`) AND re-namespaced (`iojn2_`) — the two passes + // compose; neither one replaces the other. + expect(flow.item.nodes[0].config.filter).toEqual({ status: 'stale' }); + expect(flow.item.nodes[0].config.objectName).toBe('iojn2_repair_ticket'); + }); + + it('a refused rename fails the item and names the token — never a silent legacy copy', async () => { + const conflicting = () => ({ + storable: {}, + notices: [], + conflicts: [{ + conversionId: 'flow-node-type-open-namespace', + token: 'http_request', + path: 'flows[0].nodes[0].type', + message: 'a custom executor owns this node type here', + }], + }); + const { protocol, saveMetaItem } = makeProtocol( + [legacyFlowRow], + new Map([['automation', { canonicalizeStoredFlow: conflicting }]]), + ); + + const res = await duplicate(protocol); + + expect(res.failedCount).toBe(1); + expect(res.copiedCount).toBe(0); + expect(res.failed[0].error).toContain('http_request'); + expect(res.failed[0].error).toContain('live name in this environment'); + // The point of failing: copying the un-renamed body would mint exactly + // the row this fix exists to prevent. + expect(saveMetaItem).not.toHaveBeenCalled(); + }); + + it('a flow that cannot canonicalize fails the item with the reason', async () => { + const throwing = () => { throw new Error("Unrecognized key: '_uiPosition'"); }; + const { protocol, saveMetaItem } = makeProtocol( + [legacyFlowRow], + new Map([['automation', { canonicalizeStoredFlow: throwing }]]), + ); + + const res = await duplicate(protocol); + + expect(res.failedCount).toBe(1); + expect(res.failed[0].error).toMatch(/does not canonicalize.*_uiPosition/); + expect(saveMetaItem).not.toHaveBeenCalled(); + }); + + it('no engine reachable → the flow is copied as-is, and the duplication still succeeds', async () => { + // A control-plane / metadata-only host has no automation service. The + // copy is then no better than the source row already was — but failing + // a whole duplication over it would be a regression for a deployment + // that never runs flows at all. + const { protocol, saveMetaItem } = makeProtocol([legacyFlowRow], new Map()); + + const res = await duplicate(protocol); + + expect(res).toMatchObject({ success: true, copiedCount: 1, failedCount: 0 }); + expect((saveMetaItem as any).mock.calls[0][0].item.nodes[0].config).toHaveProperty('filters'); + }); + + it('non-flow rows still go through the conversion chain', async () => { + const spy = vi.fn(canonicalizeStoredFlow); + const { protocol, saveMetaItem } = makeProtocol( + [objectRow], + new Map([['automation', { canonicalizeStoredFlow: spy }]]), + ); + await duplicate(protocol); + expect(spy).not.toHaveBeenCalled(); + const written = (saveMetaItem as any).mock.calls[0][0]; + expect(written.type).toBe('object'); + expect(written.name).toBe('iojn2_repair_ticket'); + }); + + it('an unparseable body is still reported as such, not as a canonicalization failure', async () => { + const { protocol } = makeProtocol( + [{ type: 'flow', name: 'iojn_broken', package_id: 'app.iojn', metadata: '{not json' }], + new Map([['automation', { canonicalizeStoredFlow }]]), + ); + const res = await duplicate(protocol); + expect(res.failed[0].error).toBe('unparseable metadata'); + }); +}); diff --git a/packages/metadata-protocol/src/protocol.ts b/packages/metadata-protocol/src/protocol.ts index 03e2841d52..d7e3212a6c 100644 --- a/packages/metadata-protocol/src/protocol.ts +++ b/packages/metadata-protocol/src/protocol.ts @@ -1770,6 +1770,46 @@ export class ObjectStackProtocolImplementation implements return { item, notices }; } + /** + * Resolve a flow canonicalizer from the live services registry (#4498). + * + * `convertStoredItem` skips `flow` because flow-node conversions carry + * ADR-0078's open-namespace conflict guard, which needs the automation + * engine's executor registry to tell a rename from a clobber. #4454 built + * that capability as `AutomationEngine.canonicalizeStoredFlow` and handed + * it to `migrateStoredMetadata` as an explicit hook, because the CLI has + * to boot an engine of its own to hold one. + * + * Inside a server there is nothing to thread: this protocol is constructed + * with an accessor for the kernel's service table (the same one + * `analytics` and `package` are read from), and the automation service + * registers itself under `automation`. So every caller running next to a + * live engine can have the capability for free — which is what makes the + * flow-skip fixable at `duplicatePackage` (a WRITE that was minting new + * pre-protocol rows) rather than only at the CLI. + * + * Resolution is deliberately **lazy** — per call, never cached at + * construction. Plugin init order is not guaranteed to put `automation` + * in the table before the protocol is assembled (the CLI's + * `buildDataMigrationPlugins` adds it after ObjectQL by design), and + * caching `undefined` from a too-early read would silently disable flow + * canonicalization for the life of the process. + * + * Returns `undefined` when no engine is reachable. That is a real state — + * a control-plane or metadata-only host has no automation service — and + * every caller must decide what it means for them rather than assume a + * flow was handled. + */ + private resolveFlowCanonicalizer(): + ((name: string, body: unknown) => StoredFlowCanonicalization) | undefined { + const automation = this.getServicesRegistry?.().get('automation') as + | { canonicalizeStoredFlow?: (name: string, definition: unknown) => StoredFlowCanonicalization } + | undefined; + const canonicalize = automation?.canonicalizeStoredFlow; + if (typeof canonicalize !== 'function') return undefined; + return (name, body) => canonicalize.call(automation, name, body); + } + constructor( engine: IDataEngine, getServicesRegistry?: () => Map, @@ -6441,10 +6481,12 @@ export class ObjectStackProtocolImplementation implements * * ## What it declines to touch, and says so * - * - **`flow` rows.** Flow-node conversions carry ADR-0078's open-namespace - * conflict guard, which needs the automation engine's live executor - * registry; this layer does not have it, so flows canonicalize at - * `AutomationEngine.registerFlow` and are reported `skipped` here. + * - **`flow` rows with no reachable automation engine.** Flow-node + * conversions carry ADR-0078's open-namespace conflict guard, which + * needs the engine's live executor registry. When one is reachable — + * passed as `canonicalizeFlow`, or resolved from the services registry + * (#4498) — flows are migrated like anything else (#4454); when none is, + * they are reported `skipped` with that reason, never counted done. * - **Types with no repository write path** (neither `allowOrgOverride` nor * `allowRuntimeCreate`). `saveMetaItem` routes those down the legacy * raw-engine branch, which records no history and forces `state: @@ -6464,14 +6506,21 @@ export class ObjectStackProtocolImplementation implements /** Recorded as the writer on the history + audit rows. */ actor?: string; /** - * Canonicalize a stored `flow` body (#4454). + * Canonicalize a stored `flow` body (#4454). **Optional override** — + * when omitted, the automation engine is resolved from the live + * services registry (#4498, {@link resolveFlowCanonicalizer}). + * + * `AutomationEngine.canonicalizeStoredFlow` is the implementation. + * Flow conversions carry ADR-0078's open-namespace conflict guard, + * which needs the engine's executor registry to tell a rename from a + * clobber. A caller running next to a live engine (an admin route, a + * server task) needs to pass nothing; the CLI passes its own because + * it boots an inert engine specifically to hold one, and an explicit + * hook is also what makes the flow branch testable without an engine. * - * Supplied only by a caller that holds a live automation engine — - * `AutomationEngine.canonicalizeStoredFlow` is the implementation. Flow - * conversions carry ADR-0078's open-namespace conflict guard, which - * needs the engine's executor registry to tell a rename from a clobber, - * and this layer has no way to obtain one. Omit it and flow rows are - * reported `skipped` with that reason rather than quietly counted done. + * When neither is available — a control-plane or metadata-only host — + * flow rows are reported `skipped` with that reason rather than + * quietly counted done. * * Must return the **storable** shape — conversions and the schema's * `condition` envelopes, without schema defaults. Throwing is a valid @@ -6480,6 +6529,7 @@ export class ObjectStackProtocolImplementation implements */ canonicalizeFlow?: (name: string, body: unknown) => StoredFlowCanonicalization; } = {}): Promise { + const canonicalizeFlow = request.canonicalizeFlow ?? this.resolveFlowCanonicalizer(); const apply = request.apply === true; const typeFilter = request.types && request.types.length > 0 ? new Set(request.types.map((t) => PLURAL_TO_SINGULAR[t] ?? t)) @@ -6548,22 +6598,23 @@ export class ObjectStackProtocolImplementation implements } // Flow rows need the automation engine's live executor registry for - // ADR-0078's open-namespace conflict guard, which this layer does - // not have — so they are canonicalized through a caller-supplied - // hook, or reported `skipped` when no caller can supply one (#4454). + // ADR-0078's open-namespace conflict guard — supplied by the caller + // (#4454) or resolved from the services registry (#4498), and + // reported `skipped` when neither can reach one. let flowResult: StoredFlowCanonicalization | undefined; if (singular === 'flow') { - if (!request.canonicalizeFlow) { + if (!canonicalizeFlow) { record({ ...base, outcome: 'skipped', reason: 'flows canonicalize at AutomationEngine.registerFlow — the node-type ' - + 'conflict guard needs the live executor registry this caller did not supply', + + 'conflict guard needs the live executor registry, and no automation service ' + + 'is reachable from this caller', }); continue; } try { - flowResult = request.canonicalizeFlow(base.name, body); + flowResult = canonicalizeFlow(base.name, body); } catch (e: any) { // `FlowSchema` is strict (#4001) and the region validator // hard-fails, so this is a row that cannot register at all — @@ -7662,22 +7713,91 @@ export class ObjectStackProtocolImplementation implements const copied: Array<{ type: string; name: string }> = []; const failed: Array<{ type: string; name: string; error: string }> = []; + // Resolved once for the whole copy: every flow row in this package needs + // the same engine, and a package with fifty flows should not walk the + // service table fifty times. + const canonicalizeFlow = this.resolveFlowCanonicalizer(); + for (const row of rows) { const newName = renameName(row.name); - let item: any; + const rawType = String(row.type); + const singular = PLURAL_TO_SINGULAR[rawType] ?? rawType; + let body: unknown; try { - // Canonicalize the source row before re-saving (#3903): the copy - // is a NEW write and must pass today's schema gate, so a legacy - // shape the chain owns is lifted rather than failing the copy — - // duplication never mints new rows in a pre-protocol dialect. - item = this.convertStoredItem( - String(row.type), - typeof row.metadata === 'string' ? JSON.parse(row.metadata) : (row.metadata ?? {}), - ); + body = typeof row.metadata === 'string' ? JSON.parse(row.metadata) : (row.metadata ?? {}); } catch { failed.push({ type: row.type, name: row.name, error: 'unparseable metadata' }); continue; } + + // Canonicalize the source row before re-saving (#3903): the copy is + // a NEW write and must pass today's schema gate, so a legacy shape + // the chain owns is lifted rather than failing the copy — + // duplication never mints new rows in a pre-protocol dialect. + // + // For `flow` that guarantee was false until #4498: `convertStoredItem` + // returns flows untouched, and `FlowNodeSchema.config` is an open + // `z.record`, so a pre-17 body (`delete_record` with `config.filters`) + // sailed through `saveMetaItem` and landed verbatim in a brand-new + // row. ADR-0087 justifies the whole stored-metadata design on new + // writes being canonical — "a strictly shrinking concern" — and this + // was the one live producer contradicting it. + let item: any; + if (singular === 'flow') { + if (!canonicalizeFlow) { + // No engine in this process (control-plane / metadata-only + // host). Copy the source body as-is — the honest behaviour, + // and no worse than the source row already is — rather than + // failing a duplication that has nothing to do with flows. + // `os migrate meta --stored --apply` is the finish line for + // both rows, and it reports what it could not canonicalize. + item = body; + } else { + try { + const result = canonicalizeFlow(String(row.name ?? ''), body); + if (result.conflicts.length > 0) { + // ADR-0078's guard refused a node-type rename because + // the old token is a LIVE name owned by something + // else here. Copying the un-renamed body anyway would + // mint exactly the row this fix exists to prevent, so + // the item fails and names the token (same posture as + // #4454's `failed` outcome). + const first = result.conflicts[0]!; + failed.push({ + type: row.type, + name: row.name, + error: `conversion refused — '${first.token}' at ${first.path} is a live name in ` + + `this environment (${result.conflicts.length} conflict(s)). ${first.message}`, + }); + continue; + } + item = result.storable; + } catch (e: any) { + // `FlowSchema` is strict (#4001) and the region validator + // hard-fails: this source row cannot register at all, so + // the copy would be broken the same way. Report it. + failed.push({ + type: row.type, + name: row.name, + error: `the flow does not canonicalize: ${e?.message ?? String(e)}`, + }); + continue; + } + } + } else { + try { + item = this.convertStoredItem(rawType, body); + } catch (e: any) { + // A tombstoned key throws here (ADR-0087 D2) — a genuine + // contract violation in the source, not a parse failure. + failed.push({ + type: row.type, + name: row.name, + error: `the source item does not convert: ${e?.message ?? String(e)}`, + }); + continue; + } + } const rewritten = deepRewrite(item); if (rewritten && typeof rewritten === 'object' && !Array.isArray(rewritten)) rewritten.name = newName; try { diff --git a/packages/rest/src/rest-meta-migrate-stored.test.ts b/packages/rest/src/rest-meta-migrate-stored.test.ts new file mode 100644 index 0000000000..0753394188 --- /dev/null +++ b/packages/rest/src/rest-meta-migrate-stored.test.ts @@ -0,0 +1,195 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * `POST /api/v1/meta/_migrate-stored` — the server-side form of + * `os migrate meta --stored` (#4327 / #4454 / #4498). + * + * `os migrate meta --stored` needs shell access to the deployment's database. + * A hosted operator has none, so ADR-0087's stored-metadata chain had no finish + * line at all on a managed deployment — only the per-read conversion, running + * forever. This route is that finish line, and because it runs inside a server + * that already holds a live automation engine, flow rows are covered without + * threading anything (#4498). + * + * What these pin is the route's POSTURE. The migration itself is covered by + * `metadata-protocol`'s `protocol.stored-migration.test.ts`; here the questions + * are: who may fire it, and does an under-specified request write. + */ + +import { describe, it, expect, vi } from 'vitest'; +import { RestServer } from './rest-server'; + +const REPORT = { + apply: false, + protocol: '17.0.0', + scanned: 4, + canonical: 3, + pending: 1, + rewritten: 0, + skipped: 0, + failed: 0, + rows: [], +}; + +function createMockServer() { + const noop = () => {}; + return { get: noop, post: noop, put: noop, delete: noop, patch: noop, use: noop, listen: async () => {}, close: async () => {} }; +} + +function makeRes() { + let status = 200; + const res: any = { + status: (code: number) => { status = code; return res; }, + json: (body: any) => { (res as any)._json = body; return res; }, + header: () => res, + write: () => true, + end: () => {}, + }; + return { res, getStatus: () => status, getJson: () => (res as any)._json }; +} + +/** Boot the route over a stub protocol, with `execCtx` as the resolved caller. */ +function boot(execCtx: any, protocolOverrides: Record = {}) { + const migrateStoredMetadata = vi.fn().mockResolvedValue(REPORT); + const protocol: any = { migrateStoredMetadata, ...protocolOverrides }; + const rest = new RestServer( + createMockServer() as any, + protocol as any, + { api: { requireAuth: false } } as any, + ); + (rest as any).resolveExecCtx = async () => execCtx; + rest.registerRoutes(); + const route = rest.getRoutes().find( + (r: any) => r.method === 'POST' && r.path === '/api/v1/meta/_migrate-stored', + ); + expect(route).toBeDefined(); + return { route, migrateStoredMetadata }; +} + +const run = async (route: any, body: unknown) => { + const out = makeRes(); + await route.handler({ params: {}, query: {}, body } as any, out.res); + return out; +}; + +describe('POST /meta/_migrate-stored — mounting', () => { + it('is registered BEFORE /meta/:type, so the segment is never read as a type name', async () => { + const rest = new RestServer( + createMockServer() as any, + { migrateStoredMetadata: vi.fn() } as any, + { api: { requireAuth: false } } as any, + ); + rest.registerRoutes(); + const paths = rest.getRoutes().map((r: any) => `${r.method} ${r.path}`); + expect(paths).toContain('POST /api/v1/meta/_migrate-stored'); + expect(paths.indexOf('POST /api/v1/meta/_migrate-stored')) + .toBeLessThan(paths.indexOf('GET /api/v1/meta/:type')); + }); +}); + +describe('POST /meta/_migrate-stored — capability gate', () => { + it('403s a caller without `manage_metadata`, and reads NOTHING', async () => { + const { route, migrateStoredMetadata } = boot({ userId: 'u1', systemPermissions: ['setup.access'] }); + const out = await run(route, { apply: true }); + + expect(out.getStatus()).toBe(403); + expect(out.getJson()).toMatchObject({ error: { code: 'FORBIDDEN' } }); + // Unlike the single-item `PUT /meta/:type/:name` next door, this rewrites + // every eligible row in the deployment — a session is not enough. + expect(migrateStoredMetadata).not.toHaveBeenCalled(); + }); + + it('an anonymous caller never reaches the capability gate — 401 from the meta umbrella', async () => { + // Every `/meta` route inherits the anonymous-deny wrapper + // (`registerMetadataEndpoints`), so this route is closed to anonymous + // callers by construction and the `manage_metadata` check below it is the + // second layer, not the only one. + const { route, migrateStoredMetadata } = boot(undefined); + const out = await run(route, {}); + expect(out.getStatus()).toBe(401); + expect(migrateStoredMetadata).not.toHaveBeenCalled(); + }); + + it('allows a caller holding `manage_metadata`', async () => { + const { route, migrateStoredMetadata } = boot({ userId: 'u1', systemPermissions: ['manage_metadata'] }); + const out = await run(route, {}); + expect(out.getStatus()).toBe(200); + expect(migrateStoredMetadata).toHaveBeenCalledTimes(1); + }); + + it('isSystem bypasses, matching every other capability gate', async () => { + const { route, migrateStoredMetadata } = boot({ isSystem: true }); + await run(route, {}); + expect(migrateStoredMetadata).toHaveBeenCalledTimes(1); + }); + + it('the gate fires BEFORE the protocol is probed, so 403 vs 501 leaks nothing', async () => { + // A kernel with no `migrateStoredMetadata` answers 501 to an authorized + // caller. An unauthorized one must not be able to tell the two apart. + const { route } = boot({ userId: 'u1', systemPermissions: [] }, { migrateStoredMetadata: undefined }); + const out = await run(route, {}); + expect(out.getStatus()).toBe(403); + }); +}); + +describe('POST /meta/_migrate-stored — preview by default', () => { + const admin = { userId: 'admin', systemPermissions: ['manage_metadata'] }; + + it('an empty body previews — `apply` is never inferred', async () => { + const { route, migrateStoredMetadata } = boot(admin); + await run(route, {}); + expect(migrateStoredMetadata.mock.calls[0][0].apply).toBe(false); + }); + + it('a missing body previews rather than throwing', async () => { + const { route, migrateStoredMetadata } = boot(admin); + const out = await run(route, undefined); + expect(out.getStatus()).toBe(200); + expect(migrateStoredMetadata.mock.calls[0][0].apply).toBe(false); + }); + + it('only a literal `true` applies', async () => { + const { route, migrateStoredMetadata } = boot(admin); + await run(route, { apply: 'yes' }); + expect(migrateStoredMetadata.mock.calls[0][0].apply).toBe(false); + await run(route, { apply: true }); + expect(migrateStoredMetadata.mock.calls[1][0].apply).toBe(true); + }); + + it('passes a `types` filter through, dropping non-string members', async () => { + const { route, migrateStoredMetadata } = boot(admin); + await run(route, { types: ['flow', 42, '', 'object'] }); + expect(migrateStoredMetadata.mock.calls[0][0].types).toEqual(['flow', 'object']); + }); + + it('omits `types` when none survive, so the run is not silently empty', async () => { + const { route, migrateStoredMetadata } = boot(admin); + await run(route, { types: [42] }); + expect(migrateStoredMetadata.mock.calls[0][0]).not.toHaveProperty('types'); + }); + + it('threads NO canonicalizeFlow — the protocol resolves the engine itself (#4498)', async () => { + const { route, migrateStoredMetadata } = boot(admin); + await run(route, {}); + expect(migrateStoredMetadata.mock.calls[0][0]).not.toHaveProperty('canonicalizeFlow'); + }); + + it('attributes the run to the caller — history and audit rows answer "who ran it"', async () => { + const { route, migrateStoredMetadata } = boot(admin); + await run(route, {}); + expect(migrateStoredMetadata.mock.calls[0][0].actor).toContain('admin'); + }); + + it('returns the report unwrapped', async () => { + const { route } = boot(admin); + const out = await run(route, {}); + expect(out.getJson()).toEqual(REPORT); + }); + + it('501s an authorized caller on a kernel whose protocol predates the pass', async () => { + const { route } = boot(admin, { migrateStoredMetadata: undefined }); + const out = await run(route, {}); + expect(out.getStatus()).toBe(501); + expect(out.getJson()).toMatchObject({ error: { code: 'NOT_IMPLEMENTED' } }); + }); +}); diff --git a/packages/rest/src/rest-route-ledger.ts b/packages/rest/src/rest-route-ledger.ts index 3d20e5ecc3..37b5aa031d 100644 --- a/packages/rest/src/rest-route-ledger.ts +++ b/packages/rest/src/rest-route-ledger.ts @@ -85,6 +85,8 @@ export const REST_ROUTE_LEDGER: readonly RestRouteLedgerEntry[] = [ { route: 'GET /api/v1/meta', family: 'metadata', source: 'route-manager', disposition: 'sdk', client: 'meta.getTypes' }, { route: 'GET /api/v1/meta/diagnostics', family: 'metadata', source: 'route-manager', disposition: 'sdk', client: 'meta.getDiagnostics' }, { route: 'GET /api/v1/meta/_drafts', family: 'metadata', source: 'route-manager', disposition: 'sdk', client: 'meta.listDrafts' }, + { route: 'POST /api/v1/meta/_migrate-stored', family: 'metadata', source: 'route-manager', disposition: 'sdk', client: 'meta.migrateStored', + note: 'ADR-0087 stored-row canonicalization (#4327); gated on `manage_metadata`, preview unless { apply: true }' }, { route: 'GET /api/v1/meta/:type', family: 'metadata', source: 'route-manager', disposition: 'sdk', client: 'meta.getItems' }, { route: 'GET /api/v1/meta/:type/:name/references', family: 'metadata', source: 'route-manager', disposition: 'sdk', client: 'meta.getReferences' }, { route: 'GET /api/v1/meta/book/:name/tree', family: 'metadata', source: 'route-manager', disposition: 'sdk', client: 'meta.getBookTree' }, diff --git a/packages/rest/src/rest-server.ts b/packages/rest/src/rest-server.ts index af5d843e54..c79c1e7fa1 100644 --- a/packages/rest/src/rest-server.ts +++ b/packages/rest/src/rest-server.ts @@ -2808,6 +2808,90 @@ export class RestServer { }); } + // POST /meta/_migrate-stored — rewrite stored sys_metadata rows into + // today's canonical shape (ADR-0087; #4327 / #4454 / #4498). + // + // The server-side form of `os migrate meta --stored`. The CLI form + // needs shell access to the deployment's database, which a hosted + // operator does not have — so without this route the stored-metadata + // chain has no finish line on a managed deployment, only the per-read + // conversion that runs forever. Flow rows are covered here for free: + // `migrateStoredMetadata` resolves the automation engine from the + // services registry (#4498), and a server always has a live one. + // + // Registered BEFORE `/meta/:type` so the leading-underscore segment is + // not captured as a `:type` parameter (same reason as `_drafts`). + if (metadata.endpoints.items !== false) { + this.routeManager.register({ + method: 'POST', + path: `${metaPath}/_migrate-stored`, + handler: async (req: any, res: any) => { + try { + const environmentId = isScoped ? req.params?.environmentId : undefined; + // Gate FIRST — before resolving the protocol — so an + // unauthorized caller cannot use the 501 vs 200 answer + // to probe which kernels can be migrated. + // + // This rewrites every eligible row in the deployment, + // so unlike the single-item `PUT /meta/:type/:name` it + // demands an explicit capability rather than only a + // session. `manage_metadata` is ADR-0066 D1's authoring + // capability, and a canonicalization rewrite is + // authoring; `isSystem` bypasses, matching every other + // capability gate on the platform. + const ctx = await this.resolveExecCtx(environmentId, req).catch(() => undefined); + const held = new Set( + Array.isArray(ctx?.systemPermissions) ? ctx!.systemPermissions : [], + ); + if (!ctx?.isSystem && !held.has('manage_metadata')) { + res.status(403).json({ + error: { + code: 'FORBIDDEN', + message: 'Rewriting stored metadata requires the `manage_metadata` capability.', + }, + }); + return; + } + const p = await this.resolveProtocol(environmentId, req); + if (typeof (p as any).migrateStoredMetadata !== 'function') { + res.status(501).json({ + error: { + code: 'NOT_IMPLEMENTED', + message: 'protocol.migrateStoredMetadata() is not available in this kernel', + }, + }); + return; + } + const rawTypes = (req.body as any)?.types; + const types = Array.isArray(rawTypes) + ? rawTypes.filter((t: unknown): t is string => typeof t === 'string' && t.length > 0) + : []; + // Preview by default — `apply` must be explicitly true, + // the same posture the CLI takes. A caller who sends an + // empty body gets a report and no writes. + const report = await (p as any).migrateStoredMetadata({ + apply: (req.body as any)?.apply === true, + ...(types.length > 0 ? { types } : {}), + // Attributed to the caller: this writes history + + // audit rows, and "who ran the migration" is the + // question those rows exist to answer. + actor: ctx?.userId + ? `${ctx.userId} (POST ${metadata.prefix}/_migrate-stored)` + : `POST ${metadata.prefix}/_migrate-stored`, + }); + res.json(report); + } catch (error: any) { + logError("[REST] Unhandled error:", error); + sendError(res, error); + } + }, + metadata: { + summary: 'Rewrite stored metadata rows into the canonical protocol shape', + tags: ['metadata'], + }, + }); + } + // GET /meta/:type - List items of a type if (metadata.endpoints.items !== false) { this.routeManager.register({ diff --git a/packages/runtime/src/domains/meta-migrate-stored.test.ts b/packages/runtime/src/domains/meta-migrate-stored.test.ts new file mode 100644 index 0000000000..1b7a4f5515 --- /dev/null +++ b/packages/runtime/src/domains/meta-migrate-stored.test.ts @@ -0,0 +1,159 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * `POST /meta/_migrate-stored` — the server-side form of + * `os migrate meta --stored` (#4327 / #4454 / #4498). + * + * The CLI form needs shell access to the deployment's database, which a hosted + * operator does not have, so on a managed deployment ADR-0087's stored-metadata + * chain had no finish line at all. This route is that finish line — and, unlike + * the CLI, it runs in a process that already holds a live automation engine, so + * flow rows are covered without threading anything (#4498). + * + * What is pinned here is the route's POSTURE, not the migration itself (that is + * `metadata-protocol`'s `protocol.stored-migration.test.ts`): it rewrites every + * eligible row in the deployment, so it must demand a capability rather than a + * session, and it must not write unless the caller explicitly asked it to. + */ +import { describe, it, expect, vi } from 'vitest'; +import { HttpDispatcher } from '../http-dispatcher.js'; + +const REPORT = { + apply: false, + protocol: '17.0.0', + scanned: 3, + canonical: 2, + pending: 1, + rewritten: 0, + skipped: 0, + failed: 0, + rows: [], +}; + +function make(protocol: any) { + const kernel = { + context: { + getService: (name: string) => (name === 'protocol' ? protocol : null), + }, + } as any; + return new HttpDispatcher(kernel); +} + +const ctx = (executionContext: any): any => ({ request: {}, environmentId: 'platform', executionContext }); + +describe('POST /meta/_migrate-stored — capability gate (#4327)', () => { + it('403s an authenticated caller without `manage_metadata`', async () => { + const migrateStoredMetadata = vi.fn().mockResolvedValue(REPORT); + const res = await make({ migrateStoredMetadata }).handleMetadata( + '/_migrate-stored', + ctx({ userId: 'u1', systemPermissions: ['setup.access'] }), + 'POST', + { apply: true }, + ); + expect(res.response.status).toBe(403); + // The gate is the point: an ordinary session must not be able to rewrite + // every metadata row in the deployment. + expect(migrateStoredMetadata).not.toHaveBeenCalled(); + }); + + it('allows a caller holding `manage_metadata`', async () => { + const migrateStoredMetadata = vi.fn().mockResolvedValue(REPORT); + const res = await make({ migrateStoredMetadata }).handleMetadata( + '/_migrate-stored', + ctx({ userId: 'u1', systemPermissions: ['manage_metadata'] }), + 'POST', + {}, + ); + expect(res.response.status).not.toBe(403); + expect(migrateStoredMetadata).toHaveBeenCalledTimes(1); + }); + + it('engine self-invocation (isSystem) bypasses, matching every other capability gate', async () => { + const migrateStoredMetadata = vi.fn().mockResolvedValue(REPORT); + await make({ migrateStoredMetadata }).handleMetadata( + '/_migrate-stored', + ctx({ isSystem: true }), + 'POST', + {}, + ); + expect(migrateStoredMetadata).toHaveBeenCalledTimes(1); + }); + + it('an anonymous caller is refused by the domain gate before reaching this route', async () => { + const migrateStoredMetadata = vi.fn().mockResolvedValue(REPORT); + const res = await make({ migrateStoredMetadata }).handleMetadata( + '/_migrate-stored', + ctx({}), + 'POST', + {}, + ); + expect(res.response.status).toBe(401); + expect(migrateStoredMetadata).not.toHaveBeenCalled(); + }); +}); + +describe('POST /meta/_migrate-stored — preview by default (#4327)', () => { + const admin = () => ctx({ userId: 'admin', systemPermissions: ['manage_metadata'] }); + + it('an empty body previews — `apply` is never inferred', async () => { + const migrateStoredMetadata = vi.fn().mockResolvedValue(REPORT); + await make({ migrateStoredMetadata }).handleMetadata('/_migrate-stored', admin(), 'POST', {}); + expect(migrateStoredMetadata.mock.calls[0][0].apply).toBe(false); + }); + + it('only a literal `true` applies — a truthy string does not', async () => { + const migrateStoredMetadata = vi.fn().mockResolvedValue(REPORT); + const d = make({ migrateStoredMetadata }); + await d.handleMetadata('/_migrate-stored', admin(), 'POST', { apply: 'yes' }); + expect(migrateStoredMetadata.mock.calls[0][0].apply).toBe(false); + await d.handleMetadata('/_migrate-stored', admin(), 'POST', { apply: true }); + expect(migrateStoredMetadata.mock.calls[1][0].apply).toBe(true); + }); + + it('passes a `types` filter through, dropping non-string members', async () => { + const migrateStoredMetadata = vi.fn().mockResolvedValue(REPORT); + await make({ migrateStoredMetadata }).handleMetadata( + '/_migrate-stored', admin(), 'POST', { types: ['flow', 42, '', 'object'] }, + ); + expect(migrateStoredMetadata.mock.calls[0][0].types).toEqual(['flow', 'object']); + }); + + it('omits `types` entirely when none survive, so the run is not silently empty', async () => { + const migrateStoredMetadata = vi.fn().mockResolvedValue(REPORT); + await make({ migrateStoredMetadata }).handleMetadata( + '/_migrate-stored', admin(), 'POST', { types: [42] }, + ); + expect(migrateStoredMetadata.mock.calls[0][0]).not.toHaveProperty('types'); + }); + + it('threads NO canonicalizeFlow — the protocol resolves the engine itself (#4498)', async () => { + const migrateStoredMetadata = vi.fn().mockResolvedValue(REPORT); + await make({ migrateStoredMetadata }).handleMetadata('/_migrate-stored', admin(), 'POST', {}); + expect(migrateStoredMetadata.mock.calls[0][0]).not.toHaveProperty('canonicalizeFlow'); + }); + + it('attributes the run to the caller — history and audit rows answer "who ran it"', async () => { + const migrateStoredMetadata = vi.fn().mockResolvedValue(REPORT); + await make({ migrateStoredMetadata }).handleMetadata('/_migrate-stored', admin(), 'POST', {}); + expect(migrateStoredMetadata.mock.calls[0][0].actor).toContain('admin'); + }); + + it('returns the report as-is', async () => { + const res = await make({ migrateStoredMetadata: vi.fn().mockResolvedValue(REPORT) }) + .handleMetadata('/_migrate-stored', admin(), 'POST', {}); + expect(res.response.status).toBe(200); + expect(res.response.body.data).toEqual(REPORT); + }); + + it('501s on a kernel whose protocol predates the pass', async () => { + const res = await make({}).handleMetadata('/_migrate-stored', admin(), 'POST', {}); + expect(res.response.status).toBe(501); + }); + + it('is POST-only — a GET falls through to the type-list handler, not a rewrite', async () => { + const migrateStoredMetadata = vi.fn().mockResolvedValue(REPORT); + await make({ migrateStoredMetadata, getMetaItems: vi.fn().mockResolvedValue({ items: [] }) }) + .handleMetadata('/_migrate-stored', admin(), 'GET'); + expect(migrateStoredMetadata).not.toHaveBeenCalled(); + }); +}); diff --git a/packages/runtime/src/domains/meta.ts b/packages/runtime/src/domains/meta.ts index 6944c31e15..1ef489ec5c 100644 --- a/packages/runtime/src/domains/meta.ts +++ b/packages/runtime/src/domains/meta.ts @@ -294,6 +294,62 @@ export async function handleMetadataRequest(deps: DomainHandlerDeps, path: strin return { handled: true, response: deps.error('Draft listing not supported', 501) }; } + // POST /metadata/_migrate-stored (#4327 / #4454 / #4498) + // + // The server-side entry point to the same canonicalization pass + // `os migrate meta --stored` runs. It exists because the CLI form requires + // shell access to the deployment's database, which a hosted operator does + // not have — so on a managed deployment ADR-0087's stored-metadata chain + // had no finish line at all, only the per-read conversion that never ends. + // + // Nothing about flows is threaded through here: `migrateStoredMetadata` + // resolves the automation engine from the services registry (#4498), and a + // server always has a live one — so this route covers flow rows by simply + // running in the process that owns them. + // + // Body: `{ apply?: boolean, types?: string[] }`. **Preview by default** — + // the same posture as the CLI: `apply` must be explicitly `true`, and a + // caller who sends nothing gets a report and no writes. + if (parts.length === 1 && parts[0] === '_migrate-stored' && method?.toUpperCase() === 'POST') { + // This rewrites every eligible `sys_metadata` row in the deployment, so + // unlike the single-item `PUT /metadata/:type/:name` next door it is + // gated on an explicit capability rather than on being authenticated. + // `manage_metadata` is the ADR-0066 D1 capability for authoring and + // publishing metadata, which is exactly what a rewrite is; engine + // self-invocation (`isSystem`) bypasses, matching `actionPermissionError`. + const ec: any = _context.executionContext; + if (!ec?.isSystem && !new Set(ec?.systemPermissions ?? []).has('manage_metadata')) { + return { + handled: true, + response: deps.error( + 'Rewriting stored metadata requires the `manage_metadata` capability.', + 403, + ), + }; + } + + const protocol = await deps.resolveService('protocol'); + if (!protocol || typeof (protocol as any).migrateStoredMetadata !== 'function') { + return { handled: true, response: deps.error('Stored-metadata migration not supported', 501) }; + } + const types = Array.isArray(body?.types) + ? body.types.filter((t: unknown): t is string => typeof t === 'string' && t.length > 0) + : undefined; + try { + const report = await (protocol as any).migrateStoredMetadata({ + apply: body?.apply === true, + ...(types && types.length > 0 ? { types } : {}), + // Attributed to the caller, not to the route: this writes + // history + audit rows, and "who ran the migration" is the + // question those rows exist to answer. + actor: ec?.userId ? `${ec.userId} (POST /metadata/_migrate-stored)` : 'POST /metadata/_migrate-stored', + }); + return { handled: true, response: deps.success(report) }; + } catch (e: any) { + return { handled: true, response: deps.errorFromThrown(e, 500) }; + } + } + // GET /metadata/:type (List items of type) OR /metadata/:objectName (Legacy) if (parts.length === 1) { const typeOrName = parts[0]; diff --git a/packages/runtime/src/route-ledger.ts b/packages/runtime/src/route-ledger.ts index f26d3764aa..bcd6d33b86 100644 --- a/packages/runtime/src/route-ledger.ts +++ b/packages/runtime/src/route-ledger.ts @@ -219,6 +219,8 @@ export const ROUTE_LEDGER: readonly RouteLedgerEntry[] = [ { route: 'PUT /meta/:type/:name', domain: '/meta', disposition: 'sdk', client: 'meta.saveItem' }, { route: 'GET /meta/:type/:name/published', domain: '/meta', disposition: 'sdk', client: 'meta.getPublished' }, { route: 'GET /meta/_drafts', domain: '/meta', disposition: 'sdk', client: 'meta.listDrafts' }, + { route: 'POST /meta/_migrate-stored', domain: '/meta', disposition: 'sdk', client: 'meta.migrateStored', + note: 'ADR-0087 stored-row canonicalization (#4327); gated on `manage_metadata`, preview unless { apply: true }' }, { route: 'GET /meta/objects/:name/state/:field', domain: '/meta', disposition: 'sdk', client: 'meta.getLegalNextStates' }, // ── data (legacy chain) ───────────────────────────────────────────────────