diff --git a/.changeset/migrate-meta-stored-rewrite.md b/.changeset/migrate-meta-stored-rewrite.md new file mode 100644 index 0000000000..3e90b8ec56 --- /dev/null +++ b/.changeset/migrate-meta-stored-rewrite.md @@ -0,0 +1,62 @@ +--- +"@objectstack/metadata-protocol": minor +"@objectstack/cli": patch +--- + +feat(migrate,metadata-protocol): `os migrate meta --stored` rewrites sys_metadata rows so the read-path chain has a finish line (#4327) + +#4317 closed the correctness gap from the read side: every stored-row +rehydration seam replays the full ADR-0087 conversion chain, retired entries +included, so a row written under any past protocol is *served* canonical +forever. What it deliberately did not do is make the rows themselves canonical. +A pre-17 row keeps its legacy bytes, the chain re-lowers it on every load, and +each affected row logs one conversion notice per process — deduped, but back +every boot. Until now the only things that ever rewrote such a row were a Studio +re-save and `duplicatePackage`. + +**`os migrate meta --stored`** is the pass that ends it for a deployment that +runs it. It walks `sys_metadata` — `active` and `draft`, every organization — +replays the same `applyConversionsToStoredItem` chain, and re-saves each changed +body through the normal write path, so a rewritten row gets a +`sys_metadata_history` entry, a fresh checksum and the mutation projectors, +exactly like an author's save. The history row's `source` is `migrate-stored`, +so a later diff distinguishes an upgrade from somebody's edit. + +```bash +os migrate meta --stored # preview: per-row report, writes nothing +os migrate meta --stored --apply # rewrite the rows (prompts) +os migrate meta --stored --apply --yes --json # CI / scripts +os migrate meta --stored --type view # restrict to a type (repeatable) +``` + +**Preview is the default and `--apply` is the only writing mode** — the house +rule its siblings already keep (#3617's "a dry run changes nothing"), and it +applies with more force here because what moves is metadata: every affected +row's checksum and a history entry per row. An apply run also refuses to start +while another process holds the SQLite database, for the same reason +`os migrate files-to-references --apply` does. + +**Nothing gates on this having run.** #3855's conclusion stands — an +operator-run migration cannot be relied upon, so the read path remains the +guarantee for every deployment, and no `sys_migration` flag is recorded (a flag +would advertise enforcement that does not exist). What a run buys is hygiene — +rows stop carrying pre-protocol dialects, so diffs, exports and history are +clean going forward, and the recurring notices go quiet — plus one thing that +was previously unobtainable: **an operator can assert it.** A run with nothing +left to do exits `0`, a deployment with rows still on an old dialect exits `1`, +so "my metadata is on protocol N" becomes a CI check rather than a belief. + +Three things the pass declines, and reports rather than counting as done: +`flow` rows (their seam is `AutomationEngine.registerFlow`, which holds the +executor registry the node-type conflict guard needs), types with no repository +write path (`agent` — rewriting there would record no history and force a draft +live), and rows that still fail the current schema after conversion (a genuine +contract violation the write path is right to refuse; it keeps reading through +the chain and stays fixable in Studio). + +Also new, and usable without the CLI: `protocol.migrateStoredMetadata()` returns +the same structured report an admin route would render, and `saveMetaItem` +accepts an optional `source` for the history/audit rows. `source` is not +request-derived — the REST layer builds its save request field by field and +never forwards a client-supplied value, so provenance stays something the server +states rather than something a caller claims. diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index 1bd72f351f..3a57e1e869 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -589,6 +589,7 @@ written. | `os migrate plan` | Warns and continues — a plan writes nothing either way | | `os migrate apply` | **Refuses** (exit 1, `error: database_busy` under `--json`). Stop the other process, or pass `--force` | | `os migrate files-to-references --apply` | **Refuses** likewise — it rewrites rows, so a concurrent writer is at least as dangerous | +| `os migrate meta --stored --apply` | **Refuses** likewise — it rewrites `sys_metadata` rows, and a live process saving metadata is exactly the collision | The check applies to SQLite only: Postgres and MySQL take their own server-side locks. Only same-user processes are visible without elevated privileges, and a @@ -642,6 +643,7 @@ where the data lives. |---------|-------------| | `os migrate files-to-references` | Convert legacy file-field values to `sys_file` references, verify the ownership ledger, and record the deployment's migration flag | | `os migrate value-shapes` | Scan stored reference and structured-JSON field values against the platform's value contract, and record the deployment's migration flag when clean | +| `os migrate meta --stored` | Replay the metadata conversion chain over this deployment's `sys_metadata` rows and rewrite the ones still carrying a pre-protocol shape. Hygiene, not a gate — nothing depends on it having run | ```bash os migrate files-to-references # Dry run: full report, writes nothing @@ -774,6 +776,64 @@ closed gate logs that it is enforcing, and an app that declares neither class of field says nothing at all. So a running deployment always tells you the state of its own data — which is the question `os migrate meta` cannot answer. +#### `os migrate meta --stored` + +The two commands above are about **application data**. This one is about the +**metadata itself**, at rest: the `sys_metadata` rows Studio and the runtime +authoring APIs write. + +Those rows already *read* correctly whatever protocol they were written under — +every rehydration seam replays the full conversion chain, so a body from an +older major is served in today's canonical shape and always will be. What the +rows do not do is *change*: they keep their original bytes, the chain re-lowers +them on every load, and each one logs a conversion notice once per boot. This +command ends that for the deployment that runs it. + +```bash +os migrate meta --stored # Preview: per-row report, writes nothing +os migrate meta --stored --apply # Rewrite the rows (prompts) +os migrate meta --stored --apply --yes --json # CI / scripts +os migrate meta --stored --type view --type object # Restrict to a type (repeatable) +``` + +It walks `active` and `draft` rows across every organization (archived rows are +a record of what *was* and are never read), replays the same chain the read path +does, and re-saves each changed body through the normal write path — so a +rewritten row gets a `sys_metadata_history` entry, a fresh checksum, and the +mutation projectors, exactly like an author's save. The history entry's source +is `migrate-stored`, so a later diff shows which changes were an upgrade and +which were somebody's edit. + +Three things it deliberately declines, and names in the report rather than +counting as done: + +| Not rewritten | Why | +| :--- | :--- | +| `flow` rows | Flow-node conversions carry a conflict guard that needs the automation engine's live executor registry; flows canonicalize at their own seam when the engine loads them | +| Types with no repository write path (`agent`) | Their write path records no history and would force a draft live — a half-write is worse than leaving the row to the read path | +| Rows that still fail the current schema after conversion | That is a genuine contract violation, not chain-owned history. The write path's rejection is correct; fix the row in Studio | + + +`--apply` is the only writing mode, and it rewrites **metadata** — each affected +row's checksum moves and each gets a history entry. Preview first. Like the +other row-rewriting migration, an apply run refuses to start while another +process holds the SQLite database (`--force` overrides). + + +**Nothing gates on this having run.** The read path is the guarantee, for every +deployment, whether or not anyone runs this — an operator-run migration is not +something the platform can depend on. What running it buys is hygiene (cleaner +diffs, exports and history from here on, and the recurring boot notices go +quiet) plus one thing that was previously unobtainable: **you can assert it.** +A run with nothing left to do exits `0`; a deployment with rows still carrying +an old dialect exits `1`. So "my metadata is on protocol N" becomes a check +rather than a belief. + +Note the division of labour with the default mode: `os migrate meta --from N` +rewrites an **author's source** and reads no database; `--stored` rewrites **one +deployment's rows** and reads no config. Same chain, opposite ends of the +contract — which is why the two modes are mutually exclusive. + ### Scaffolding | Command | Alias | Description | diff --git a/content/docs/releases/v17.mdx b/content/docs/releases/v17.mdx index 3c3ac3da59..a5b9e5ebee 100644 --- a/content/docs/releases/v17.mdx +++ b/content/docs/releases/v17.mdx @@ -1672,7 +1672,12 @@ platform capabilities an administrator gains, then the Console delta. source at every read seam (including `registerFlow` rehydration) — a stored action with the removed `execute` dispatches via `target` again; boot hydration validates each row post-conversion and diagnoses invalid ones - with `[metadata_spec_invalid]` instead of shrugging. + with `[metadata_spec_invalid]` instead of shrugging. **The rows themselves + can now be brought forward too (#4327):** `os migrate meta --stored` + replays the chain over `sys_metadata` and rewrites what still carries a + pre-protocol shape, through the normal write path. Optional — the read path + is the guarantee either way — but it is what makes the conversion pass a + no-op on your data instead of a permanent shim. - **Studio's metadata forms tell the truth (#3786).** Four of seventeen forms had drifted from their schemas, so controls saved nothing: the Object → Capabilities toggles bound a key the schema does not declare (all seven @@ -1977,6 +1982,12 @@ covers are folded into the list below rather than left to the changelog.) fix what it reports before `--apply`. It converts nothing — the values it names are application data — and a scan that was truncated or could not read an object fails the gate even at zero violations. +- **Stored metadata (optional):** run `os migrate meta --stored` to see which + `sys_metadata` rows still carry a pre-17 shape, and `--apply` to rewrite + them. Unlike the two above this opens no gate and nothing depends on it — + those rows already read canonically, forever. It stops them re-converting on + every load, keeps diffs and exports clean going forward, and gives you an + exit code to assert on: nothing left to do exits `0`. - **Datasources:** verify every declared datasource connects in every environment — a bound datasource that cannot connect now fails the boot instead of failing every later query. diff --git a/docs/adr/0087-metadata-protocol-upgrade-contract.md b/docs/adr/0087-metadata-protocol-upgrade-contract.md index d11fe813d2..9c98420c8f 100644 --- a/docs/adr/0087-metadata-protocol-upgrade-contract.md +++ b/docs/adr/0087-metadata-protocol-upgrade-contract.md @@ -421,3 +421,39 @@ diverged (#3903). This addendum extends the contract to data at rest: at boot would unhook live tables and make the row unfixable in Studio (availability over purity for data at rest; the same verdict reaches Studio as `_diagnostics` on every read). + +## Addendum (2026-08-01) — the stored chain gets a finish line (#4327) + +The addendum above makes a legacy row read canonical *forever*, which is the +correctness guarantee — and, read literally, also a promise that the chain runs +on that row forever. `os migrate meta --stored` +(`ObjectStackProtocolImplementation.migrateStoredMetadata`) lets a deployment +end that for itself: it walks `sys_metadata` (active + draft, all orgs), replays +the same `applyConversionsToStoredItem` pass, and re-saves each changed body +through `saveMetaItem` with `source: 'migrate-stored'` — history row, checksum, +mutation projectors and all. Preview is the default; `--apply` is the only +writing mode. + +- **Not load-bearing, and no flag.** #3855's conclusion stands: an operator-run + migration cannot be relied on, so the read path — not this — remains the + guarantee, and nothing gates on it having run. Deliberately no `sys_migration` + row either: unlike ADR-0104's two gates, a flag here would advertise + enforcement that does not exist. The verifiable statement operators wanted is + the **re-run** — a second pass reporting every row canonical exits 0, so "my + metadata is on protocol N" is a check rather than a belief. +- **The write path's gate is not bypassed.** A body that still fails the current + schema after conversion is refused (422) and reported, exactly as the bullet + above describes for reads: it is a genuine contract violation, and the pass + has no more standing to persist it than an author does. It keeps reading + through the chain and stays fixable in Studio. +- **The version layer stays verbatim.** `sys_metadata_history` is appended to, + never rewritten. Canonicalizing a past version's body would break the + checksum↔body pairing this contract depends on — the migration is a new + commit, not a rewrite of history. +- **What the pass does not cover, it names.** Flows (their seam is + `AutomationEngine.registerFlow`, which holds the executor registry the + conflict guard needs) and types with no repository write path are reported as + `skipped` with the reason, never counted as done. Giving flows the same finish + line needs a canonicalization entry point on the automation engine — tracked + as #4454, and worth doing precisely because the graduated flow-node + conversions are where the most stored dialect lives. diff --git a/packages/cli/package.json b/packages/cli/package.json index f300cf4b54..253ad9351d 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -55,6 +55,7 @@ "@objectstack/lint": "workspace:*", "@objectstack/mcp": "workspace:*", "@objectstack/metadata": "workspace:*", + "@objectstack/metadata-protocol": "workspace:*", "@objectstack/objectql": "workspace:^", "@objectstack/observability": "workspace:^", "@objectstack/platform-objects": "workspace:*", diff --git a/packages/cli/src/commands/migrate/meta.stored-flags.test.ts b/packages/cli/src/commands/migrate/meta.stored-flags.test.ts new file mode 100644 index 0000000000..c5f860b8c7 --- /dev/null +++ b/packages/cli/src/commands/migrate/meta.stored-flags.test.ts @@ -0,0 +1,68 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #4327 — the flag surface where `os migrate meta`'s two modes meet. + * + * One command, two subjects: `--from N` replays the ADR-0087 chain over an + * **author's source** (a config file, no database in reach), `--stored` replays + * it over **one deployment's** `sys_metadata` rows (a database, no config in + * reach). Everything below pins a boundary that a plausible-looking oclif + * declaration gets wrong. + */ +import { describe, expect, it } from 'vitest'; +import MigrateMeta, { storedOnlyFlagsIn } from './meta.js'; + +describe('storedOnlyFlagsIn (#4327)', () => { + it('reports the stored-only flags the operator typed', () => { + expect(storedOnlyFlagsIn(['--from', '16', '--apply'])).toEqual(['apply']); + expect(storedOnlyFlagsIn(['--from=16', '--type=view', '--force'])).toEqual(['force', 'type']); + expect(storedOnlyFlagsIn(['--from', '16', '-y'])).toEqual(['yes']); + }); + + it('says nothing about a run that typed none of them', () => { + expect(storedOnlyFlagsIn(['--from', '16', '--step', '--json'])).toEqual([]); + expect(storedOnlyFlagsIn([])).toEqual([]); + }); + + it('never double-reports --yes given both spellings', () => { + expect(storedOnlyFlagsIn(['--stored', '--yes', '-y'])).toEqual(['yes']); + }); + + it('reads argv, not the environment — an exported OS_DATABASE_URL is not a typed flag', () => { + // The trap that made `dependsOn` unusable: oclif fills `--database-url` + // from `OS_DATABASE_URL`, so a merely-exported env var would have counted + // as "provided" and broken `os migrate meta --from N` for anyone who has + // one set. Provenance comes from argv precisely so it cannot. + expect(storedOnlyFlagsIn(['--from', '16'])).toEqual([]); + expect(storedOnlyFlagsIn(['--stored', '--database-url', 'sqlite://x.db'])).toEqual(['database-url']); + }); +}); + +describe('MigrateMeta flag declarations (#4327)', () => { + const flags = MigrateMeta.flags as Record; + + it('does not declare --from required — that would reject every --stored run', () => { + expect(flags.from.required).not.toBe(true); + }); + + it('makes the authored-chain flags exclusive with --stored', () => { + // `--from` names a major an author wrote against; a stored row carries its + // own history and gets the full chain regardless. Accepting both would + // imply the stored pass honours a range it does not have. + for (const name of ['from', 'to', 'step', 'out']) { + expect(flags[name].exclusive).toContain('stored'); + } + }); + + it('leaves the stored-only flags free of dependsOn', () => { + // Enforced by `storedOnlyFlagsIn` instead: see the env-var case above. + for (const name of ['apply', 'yes', 'force', 'type', 'database-url']) { + expect(flags[name].dependsOn).toBeUndefined(); + } + }); + + it('defaults --apply off — preview is the only mode a bare run can have', () => { + expect(flags.apply.default).toBe(false); + expect(flags.stored.default).toBe(false); + }); +}); diff --git a/packages/cli/src/commands/migrate/meta.ts b/packages/cli/src/commands/migrate/meta.ts index f9c0f81a3b..a5a4aaa17f 100644 --- a/packages/cli/src/commands/migrate/meta.ts +++ b/packages/cli/src/commands/migrate/meta.ts @@ -3,6 +3,7 @@ import { Args, Command, Flags } from '@oclif/core'; import { writeFileSync } from 'node:fs'; import { resolve } from 'node:path'; +import { createInterface } from 'node:readline'; import chalk from 'chalk'; import { ObjectStackDefinitionSchema, @@ -25,10 +26,45 @@ import { createTimer, emitJson, } from '../../utils/format.js'; +import { bootSchemaStack } from '../../utils/schema-migrate.js'; +import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js'; +import { OCCUPANCY_HINT, probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js'; +import { describeOccupancy } from '../../utils/sqlite-occupancy.js'; + +async function confirm(question: string): Promise { + if (!process.stdin.isTTY) return false; // non-interactive → require --yes + const rl = createInterface({ input: process.stdin, output: process.stdout }); + try { + const answer: string = await new Promise((res) => rl.question(question, res)); + return /^y(es)?$/i.test(answer.trim()); + } finally { + rl.close(); + } +} /** The protocol major that introduced the per-deployment value-shape gates. */ const VALUE_SHAPE_GATE_MAJOR = 17; +/** Flags that mean something only in `--stored` mode (#4327). */ +const STORED_ONLY_FLAGS = ['apply', 'yes', 'force', 'type', 'database-url'] as const; + +/** + * Which stored-only flags the operator actually TYPED. + * + * oclif's own `dependsOn` cannot answer this: a boolean with `default: false` + * and an `env`-backed string both read as "provided" to it, so declaring + * `dependsOn: ['stored']` on `--database-url` would make a merely-exported + * `OS_DATABASE_URL` break `os migrate meta --from N`. The raw argv is the only + * signal for intent, so the guard reads that. + */ +export function storedOnlyFlagsIn(argv: readonly string[]): string[] { + const typed = STORED_ONLY_FLAGS.filter((f) => + argv.some((a) => a === `--${f}` || a.startsWith(`--${f}=`)), + ) as string[]; + if (argv.includes('-y') && !typed.includes('yes')) typed.push('yes'); + return typed; +} + interface PendingDataMigration { /** `sys_migration` row id the run records. */ id: string; @@ -120,16 +156,35 @@ function printPendingDataMigrations(pending: PendingDataMigration[]): void { * unsafe and lossy); `--out` writes the canonicalized stack as a JSON snapshot * the agent can diff and adopt. `--step` prints a per-hop checkpoint so a failure * can be bisected to the exact major. + * + * ## `--stored`: the same chain, over data at rest (#4327) + * + * The default mode above has one subject — the **author's source**, read from a + * config file, with no database in reach. `--stored` has the other: the + * `sys_metadata` rows of **one deployment**. Same chain, same canonical target, + * opposite ends of the contract, which is why they share a command rather than + * splitting into two that would both be called "migrate the metadata". + * + * They are mutually exclusive for the same reason: `--from` describes a + * protocol major an author wrote against, and a stored row already carries its + * own history — the stored pass replays the full chain (retired entries + * included) because a row at rest has no author to ask. See + * {@link MigrateMeta.runStored}. */ export default class MigrateMeta extends Command { static override description = - 'Replay the metadata protocol migration chain from a past major to current (ADR-0087 D3).'; + 'Replay the metadata protocol migration chain from a past major to current (ADR-0087 D3). ' + + 'With --stored, replay it over this deployment\'s sys_metadata rows instead of an authored config.'; static override examples = [ '$ os migrate meta --from 10', '$ os migrate meta --from 10 --step', '$ os migrate meta --from 11 --to 12 --json', '$ os migrate meta --from 10 --out migrated.stack.json', + '$ os migrate meta --stored', + '$ os migrate meta --stored --apply', + '$ os migrate meta --stored --apply --yes --json', + '$ os migrate meta --stored --type view --type object', ]; static override args = { @@ -138,23 +193,97 @@ export default class MigrateMeta extends Command { static override flags = { from: Flags.integer({ - description: 'The protocol major the metadata was authored against.', - required: true, + description: 'The protocol major the metadata was authored against (required without --stored).', + exclusive: ['stored'], }), to: Flags.integer({ description: `Target protocol major (defaults to this runtime's, ${PROTOCOL_MAJOR}).`, + exclusive: ['stored'], }), step: Flags.boolean({ description: 'Print a per-hop checkpoint (for per-major verify / bisection).', default: false, + exclusive: ['stored'], + }), + out: Flags.string({ + description: 'Write the migrated stack as a JSON snapshot to this path.', + exclusive: ['stored'], + }), + stored: Flags.boolean({ + description: + "Canonicalize this deployment's sys_metadata rows in place instead of an authored config " + + '(read-only preview unless --apply).', + default: false, + }), + 'database-url': Flags.string({ + description: '--stored: database to canonicalize (defaults to $OS_DATABASE_URL / the project DB)', + env: 'OS_DATABASE_URL', + }), + apply: Flags.boolean({ + description: '--stored: rewrite the rows (default is a read-only preview)', + default: false, + }), + yes: Flags.boolean({ + char: 'y', + description: '--stored: skip the --apply confirmation prompt', + default: false, + }), + force: Flags.boolean({ + description: '--stored: apply even when another process is using the database (SQLite occupancy check)', + default: false, + }), + type: Flags.string({ + description: '--stored: restrict to this metadata type (repeatable; default: every type)', + multiple: true, }), - out: Flags.string({ description: 'Write the migrated stack as a JSON snapshot to this path.' }), json: Flags.boolean({ description: 'Output the machine-readable migration result as JSON.' }), }; async run(): Promise { const { args, flags } = await this.parse(MigrateMeta); const timer = createTimer(); + + if (flags.stored) { + await this.runStored(flags, timer); + return; + } + + // A stored-only flag typed without `--stored` is refused rather than + // ignored: `--apply` in particular reads as "and write it", and the + // authored-source mode has nothing to write to. + const typed = storedOnlyFlagsIn(this.argv); + if (typed.length > 0) { + const message = + `${typed.map((f) => `--${f}`).join(', ')} only appl${typed.length > 1 ? 'y' : 'ies'} to ` + + '`os migrate meta --stored` (the pass over a deployment\'s sys_metadata rows). ' + + 'The authored-source chain reads a config file and writes nothing but --out.'; + if (flags.json) { + await emitJson({ error: 'stored_only_flag', flags: typed, message }, 0, { compact: true }); + this.exit(1); + return; + } + printError(message); + this.exit(1); + return; + } + + // `--from` is required for the authored-source chain and meaningless for + // `--stored` (a row carries its own history), so it is validated here + // rather than declared `required` — oclif would reject `--stored` runs. + if (flags.from === undefined) { + const message = + 'Missing required flag --from (the protocol major your metadata was authored against). ' + + 'To canonicalize a deployment\'s stored rows instead, run `os migrate meta --stored`.'; + if (flags.json) { + await emitJson({ error: 'missing_from_major', message }, 0, { compact: true }); + this.exit(1); + return; + } + printError(message); + this.exit(1); + return; + } + const fromMajor = flags.from; const toMajor = flags.to ?? PROTOCOL_MAJOR; if (!flags.json) printHeader('Migrate · meta'); @@ -169,13 +298,13 @@ export default class MigrateMeta extends Command { // D2 pass. Running the D2 pass here would leave the chain's diff empty. const normalized = normalizeStackInput(config as Record, { convert: false }); - if (!flags.json) printStep(`Replaying chain: protocol ${flags.from} → ${toMajor}…`); - const result = applyMetaMigrations(normalized, flags.from, toMajor); + if (!flags.json) printStep(`Replaying chain: protocol ${fromMajor} → ${toMajor}…`); + const result = applyMetaMigrations(normalized, fromMajor, toMajor); // Prove the migrated stack is schema-valid — the "generated, provably valid // diff" the consumer agent reviews (ADR-0087 D3/D5). const parsed = ObjectStackDefinitionSchema.safeParse(result.stack); - const specChanges = composeSpecChanges(flags.from, toMajor); + const specChanges = composeSpecChanges(fromMajor, toMajor); const dataMigrations = pendingDataMigrations(result.stack, result.fromMajor, result.toMajor); if (flags.json) { @@ -205,7 +334,7 @@ export default class MigrateMeta extends Command { } printInfo(`Config: ${chalk.white(absolutePath)}`); - printInfo(`Chain: protocol ${flags.from} → ${toMajor} (runtime ${PROTOCOL_VERSION})`); + printInfo(`Chain: protocol ${fromMajor} → ${toMajor} (runtime ${PROTOCOL_VERSION})`); console.log(''); if (result.applied.length === 0 && result.todos.length === 0) { @@ -281,4 +410,186 @@ export default class MigrateMeta extends Command { this.exit(1); } } + + /** + * `os migrate meta --stored` — canonicalize this deployment's `sys_metadata` + * rows so the read-path conversion chain has a finish line (#4327). + * + * #4317 made every stored-row rehydration seam replay the full ADR-0087 chain, + * so a row written under any past protocol *reads* canonical forever. The rows + * stayed legacy, though: the chain re-lowers them on every load and each one + * warns once per boot. This run ends that — same chain, same policy, but the + * result is written back through the normal write path (history row, checksum, + * mutation projectors) with `source: 'migrate-stored'`. + * + * **Preview by default; `--apply` is the only writing mode.** That is the + * house rule its two siblings already keep (`os migrate value-shapes`, + * `os migrate files-to-references`, #3617's "a dry run changes nothing"), and + * the reason applies with more force here: this rewrites *metadata*, so a + * surprise run would move every affected row's checksum and mint a history + * entry against each. + * + * Nothing gates on this having run — #3855's conclusion that operator-run + * migrations cannot be relied on still holds, and the read path stays the + * guarantee. What a run buys is hygiene plus one thing that was previously + * unobtainable: **an operator can now assert it.** A second pass reporting + * every row canonical exits 0; a deployment with work left exits 1, so "my + * metadata is on protocol N" becomes a CI check instead of a belief. + */ + private async runStored( + flags: { + json: boolean; + apply: boolean; + yes: boolean; + force: boolean; + type?: string[]; + 'database-url'?: string; + }, + timer: { elapsed: () => number; display: () => string }, + ): Promise { + const apply = flags.apply; + if (!flags.json) printHeader('Migrate · meta --stored'); + + // Occupancy gate — an apply run rewrites rows, so a live writer on the same + // SQLite file is the same hazard `os migrate files-to-references --apply` + // refuses for. Probed BEFORE boot (afterwards our own pool is what the probe + // finds) and before the prompt, so nobody confirms something we then refuse. + const occupancy = await probeMigrationTarget(flags['database-url']); + if (occupancy.status === 'busy' && apply && !flags.force) { + if (flags.json) { + await emitJson({ + error: 'database_busy', + database: occupancy.filename, + signal: occupancy.signal, + detail: occupancy.detail, + hint: OCCUPANCY_HINT, + }, 0, { compact: true }); + this.exit(1); + return; + } + printError(describeOccupancy(occupancy)); + printWarning(OCCUPANCY_HINT); + this.exit(1); + return; + } + if (occupancy.status === 'busy' && !flags.json) { + printWarning(apply + ? `--force: ${describeOccupancy(occupancy)} Rewriting anyway — the live process may save metadata mid-run.` + : `${describeOccupancy(occupancy)} The preview below writes nothing, but a live process may ` + + 'be saving metadata while it runs.'); + } + if (occupancy.status === 'unknown' && !flags.json) { + printWarning(`Could not check whether the database is in use — ${occupancy.detail}`); + } + + if (apply && !flags.yes) { + if (flags.json || !process.stdin.isTTY) { + if (flags.json) { + await emitJson({ error: 'confirmation_required', hint: 'pass --yes' }, 0, { compact: true }); + this.exit(1); + return; + } + printWarning( + 'Apply mode rewrites sys_metadata rows in place — each rewritten row gets a new checksum ' + + 'and a history entry. Re-run with --yes to confirm, or run without --apply to preview.', + ); + this.exit(1); + return; + } + const ok = await confirm( + chalk.bold('\nRewrite the stored metadata rows that carry a pre-protocol shape? [y/N] '), + ); + if (!ok) { + printInfo('Aborted — nothing written.'); + return; + } + } + + if (!flags.json) { + printStep(apply ? 'Booting data stack (APPLY mode)…' : 'Booting data stack (preview only)…'); + } + + let stack; + try { + // `PlatformObjectsPlugin` only — this pass needs `sys_metadata` and its + // history/audit siblings, which the protocol assembly registers itself. + // No storage adapter: unlike the file migration, nothing here reads bytes. + stack = await bootSchemaStack({ + ...(flags['database-url'] ? { databaseUrl: flags['database-url'] } : {}), + extraPlugins: await buildDataMigrationPlugins(), + }); + } catch (error: any) { + if (flags.json) { await emitJson({ error: error.message }, 0, { compact: true }); this.exit(1); return; } + printError(error.message || String(error)); + this.exit(1); + return; + } + + // Collected rather than thrown: `this.exit()` raises an oclif ExitError, + // which the catch below would then report as a bare "EEXIT: 1" over the + // real message. Decide the code here, exit after the stack is down. + let exitCode = 0; + try { + const protocol: any = stack.kernel.getService('protocol'); + if (typeof protocol?.migrateStoredMetadata !== 'function') { + throw new Error( + 'No metadata protocol on this stack — cannot walk sys_metadata. ' + + 'Run this from a project root whose stack registers the ObjectQL engine.', + ); + } + + const { formatStoredMigrationReport, storedMigrationClean } = + await import('@objectstack/metadata-protocol'); + + const report = await protocol.migrateStoredMetadata({ + apply, + ...(flags.type && flags.type.length > 0 ? { types: flags.type } : {}), + actor: 'os migrate meta --stored', + }); + const clean = storedMigrationClean(report); + if (!clean) exitCode = 1; + + if (flags.json) { + await emitJson({ database: stack.dbLabel, ...report, clean, duration: timer.elapsed() }); + } else { + printInfo(`Database: ${chalk.white(stack.dbLabel)}`); + console.log(''); + console.log(formatStoredMigrationReport(report).join('\n')); + console.log(''); + + if (report.scanned === 0) { + // Exits 0 — nothing is wrong — but does not claim a clean bill for a + // database it never read a row from. + printInfo(`No stored metadata to examine ${chalk.dim(`(${timer.display()})`)}`); + } else if (clean && apply) { + printSuccess( + `Stored metadata is on protocol ${report.protocol} — rewrote ${report.rewritten} row(s) ` + + `${chalk.dim(`(${timer.display()})`)}`, + ); + } else if (clean) { + printSuccess( + `Stored metadata is already on protocol ${report.protocol} — nothing to rewrite ` + + `${chalk.dim(`(${timer.display()})`)}`, + ); + } else if (report.failed > 0) { + printError( + `${report.failed} row(s) could not be rewritten. They keep reading canonically through the ` + + 'chain, so nothing is broken — but their stored bytes stay legacy until the reason above is fixed.', + ); + } else { + printWarning( + `${report.pending} row(s) carry a pre-protocol shape. They read canonically today (the chain ` + + 'runs on every load); re-run with --apply to persist it.', + ); + } + } + } catch (error: any) { + exitCode = 1; + if (flags.json) await emitJson({ error: error.message }, 0, { compact: true }); + else printError(error.message || String(error)); + } finally { + await stack.shutdown(); + } + if (exitCode !== 0) this.exit(exitCode); + } } diff --git a/packages/metadata-protocol/src/index.ts b/packages/metadata-protocol/src/index.ts index e7e28d01bb..303677250a 100644 --- a/packages/metadata-protocol/src/index.ts +++ b/packages/metadata-protocol/src/index.ts @@ -15,6 +15,14 @@ export type { ExtendedOperation, } from './sys-metadata-repository.js'; +export { formatStoredMigrationReport, storedMigrationClean } from './stored-migration.js'; +export type { + StoredMigrationNotice, + StoredMigrationOutcome, + StoredMigrationReport, + StoredMigrationRow, +} from './stored-migration.js'; + export { computeMetadataDiagnostics, computeViewReferenceDiagnostics, diff --git a/packages/metadata-protocol/src/protocol.stored-migration.test.ts b/packages/metadata-protocol/src/protocol.stored-migration.test.ts new file mode 100644 index 0000000000..8155d82d9c --- /dev/null +++ b/packages/metadata-protocol/src/protocol.stored-migration.test.ts @@ -0,0 +1,427 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #4327 — `os migrate meta --stored`: the read-path conversion chain gets a + * finish line. + * + * #3903/#4317 made every stored-row rehydration seam replay the full chain, so + * a legacy row *reads* canonical forever. It stayed legacy on disk, though — + * re-lowered on every load, warning once per boot. `migrateStoredMetadata` + * writes the canonical body back through the normal write path so the row + * itself stops carrying the old dialect. + * + * Rows are seeded straight into the stub engine, deliberately bypassing + * `saveMetaItem`'s schema gate — exactly like a real row written years ago + * under an older protocol. What the pass then does with them is the contract + * under test: preview writes nothing, apply rewrites through the repository + * (history row + fresh checksum + `source: 'migrate-stored'`), and everything + * it declines to touch says so instead of being silently counted clean. + */ +import { describe, expect, it } from 'vitest'; +import { ObjectStackProtocolImplementation } from './protocol.js'; +import { formatStoredMigrationReport, storedMigrationClean } from './stored-migration.js'; + +interface Row { + id: string; + type: string; + name: string; + organization_id: string | null; + package_id: string | null; + state: string; + checksum: string | null; + metadata: string; +} + +function matches(r: Record, where: Record): boolean { + for (const [k, v] of Object.entries(where)) { + if (v === undefined) continue; + if ((r[k] ?? null) !== v) return false; + } + return true; +} + +/** + * A multi-table stub engine: `sys_metadata` seeded from `seedRows`, every other + * table (`sys_metadata_history`, `sys_metadata_audit`) created on first write. + * The repository write path needs all of them, and the history table is where + * this feature's central claim — "re-saved through the normal write path" — is + * actually observable. + */ +function makeStubEngine( + seedRows: Array & { type: string; name: string; metadata: unknown }>, +) { + let nextId = 0; + const tables = new Map[]>(); + tables.set( + 'sys_metadata', + seedRows.map((r) => ({ + id: `r_${++nextId}`, + organization_id: null, + package_id: null, + state: 'active', + checksum: `sha256:seed_${nextId}`, + ...r, + metadata: typeof r.metadata === 'string' ? r.metadata : JSON.stringify(r.metadata), + })), + ); + const rowsOf = (t: string): Record[] => { + let rows = tables.get(t); + if (!rows) tables.set(t, (rows = [])); + return rows; + }; + + const engine: any = { + async find(t: string, opts?: { where?: Record }) { + return rowsOf(t).filter((r) => matches(r, opts?.where ?? {})); + }, + async findOne(t: string, opts?: { where?: Record }) { + return rowsOf(t).find((r) => matches(r, opts?.where ?? {})) ?? null; + }, + async insert(t: string, row: Record) { + const withId = { id: row.id ?? `r_${++nextId}`, ...row }; + rowsOf(t).push(withId); + return withId; + }, + async update(t: string, patch: Record, opts: { where: Record }) { + const target = rowsOf(t).find((r) => matches(r, opts.where)); + if (target) Object.assign(target, patch); + return target ?? { id: 'x' }; + }, + async delete() { return { deleted: 0 }; }, + registry: { + listItems: () => [], + isPackageDisabled: () => false, + registerItem: () => { /* no-op */ }, + registerObject: () => { /* no-op */ }, + }, + }; + return { engine, tables }; +} + +const metaRows = (tables: Map[]>) => tables.get('sys_metadata')!; +const historyRows = (tables: Map[]>) => + tables.get('sys_metadata_history') ?? []; + +/** + * A protocol-≤16 object row: `conditionalRequired` was removed from the spec in + * 17 (#3855) and its conversion is `retiredFromLoadPath` — the authored load + * seam refuses it, the stored seam keeps lowering it, and this pass persists + * the lowering. + */ +const legacyObjectRow = { + type: 'object', + name: 'crm_invoice', + metadata: { + name: 'crm_invoice', + label: 'Invoice', + fields: { + status: { type: 'select', label: 'Status' }, + amount: { type: 'currency', label: 'Amount', conditionalRequired: "record.status == 'sent'" }, + }, + }, +}; + +/** The same object, already canonical — the shape every row ends up in. */ +const canonicalObjectRow = { + type: 'object', + name: 'crm_quote', + metadata: { + name: 'crm_quote', + label: 'Quote', + fields: { + status: { type: 'select', label: 'Status' }, + amount: { type: 'currency', label: 'Amount', requiredWhen: "record.status == 'sent'" }, + }, + }, +}; + +/** A pre-17 standalone action row still carrying the removed `execute` alias. */ +const legacyActionRow = { + type: 'action', + name: 'convert', + metadata: { name: 'convert', label: 'Convert', type: 'script', object: 'crm_invoice', execute: 'convertHandler' }, +}; + +describe('migrateStoredMetadata — preview (#4327)', () => { + it('reports the rows carrying a pre-protocol shape and writes nothing', async () => { + const { engine, tables } = makeStubEngine([legacyObjectRow, canonicalObjectRow]); + const before = JSON.stringify(metaRows(tables)); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata(); + + expect(report.apply).toBe(false); + expect(report.scanned).toBe(2); + expect(report.pending).toBe(1); + expect(report.canonical).toBe(1); + expect(report.rewritten).toBe(0); + // The whole point of a preview: the bytes are exactly as they were, and + // no history row was appended either. + expect(JSON.stringify(metaRows(tables))).toBe(before); + expect(historyRows(tables)).toHaveLength(0); + }); + + it('names the conversion per row, so a preview is actionable rather than a count', async () => { + const { engine } = makeStubEngine([legacyObjectRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata(); + + expect(report.rows).toHaveLength(1); + const row = report.rows[0]!; + expect(row).toMatchObject({ type: 'object', name: 'crm_invoice', outcome: 'pending', state: 'active' }); + expect(row.notices.length).toBeGreaterThan(0); + expect(row.notices[0]!.from).toBe('conditionalRequired'); + expect(row.notices[0]!.to).toBe('requiredWhen'); + // An already-canonical row is counted, never itemised — otherwise a + // healthy deployment's report is a wall of rows that need nothing. + expect(report.rows.every((r) => r.outcome !== 'canonical')).toBe(true); + }); + + it('is not "clean" while work remains — that verdict is what a CI gate reads', async () => { + const { engine } = makeStubEngine([legacyObjectRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + expect(storedMigrationClean(await protocol.migrateStoredMetadata())).toBe(false); + }); +}); + +describe('migrateStoredMetadata — apply (#4327)', () => { + it('rewrites the row in place with the canonical body', async () => { + const { engine, tables } = makeStubEngine([legacyObjectRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.rewritten).toBe(1); + expect(report.pending).toBe(0); + expect(storedMigrationClean(report)).toBe(true); + + const stored = JSON.parse(metaRows(tables)[0]!.metadata); + expect(stored.fields.amount.requiredWhen).toBe("record.status == 'sent'"); + expect('conditionalRequired' in stored.fields.amount).toBe(false); + }); + + it('goes through the normal write path — history row, fresh checksum, migrate-stored source', async () => { + const { engine, tables } = makeStubEngine([legacyObjectRow]); + const seededChecksum = metaRows(tables)[0]!.checksum; + const protocol = new ObjectStackProtocolImplementation(engine); + + await protocol.migrateStoredMetadata({ apply: true }); + + const history = historyRows(tables); + expect(history).toHaveLength(1); + expect(history[0]).toMatchObject({ + type: 'object', + name: 'crm_invoice', + source: 'migrate-stored', + // The lineage is intact: the new version's parent is the row's + // pre-migration checksum, not a null "created from nothing". + previous_checksum: seededChecksum, + }); + // A rewritten body is a new content hash — the row is no longer + // addressed by the checksum its legacy bytes had. + expect(metaRows(tables)[0]!.checksum).not.toBe(seededChecksum); + expect(metaRows(tables)[0]!.checksum).toBe(history[0]!.checksum); + }); + + it('re-running is a no-op — the second pass finds every row canonical', async () => { + const { engine, tables } = makeStubEngine([legacyObjectRow, legacyActionRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const first = await protocol.migrateStoredMetadata({ apply: true }); + expect(first.rewritten).toBe(2); + + const second = await protocol.migrateStoredMetadata({ apply: true }); + expect(second.scanned).toBe(2); + expect(second.canonical).toBe(2); + expect(second.rewritten).toBe(0); + expect(second.rows).toHaveLength(0); + // Idempotence is the operator's verifiable statement: nothing new was + // written on the second run either. + expect(historyRows(tables)).toHaveLength(2); + }); + + it('rewrites a DRAFT row as a draft — the pass never promotes staged work live', async () => { + const { engine, tables } = makeStubEngine([{ ...legacyObjectRow, state: 'draft' }]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.rewritten).toBe(1); + expect(report.rows[0]).toMatchObject({ state: 'draft', outcome: 'rewritten' }); + expect(metaRows(tables)[0]!.state).toBe('draft'); + }); + + it('walks every org, not just the env-wide bucket', async () => { + const { engine, tables } = makeStubEngine([ + legacyObjectRow, + { ...legacyActionRow, organization_id: 'org_a' }, + ]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.scanned).toBe(2); + expect(report.rewritten).toBe(2); + const orgRow = metaRows(tables).find((r) => r.organization_id === 'org_a')!; + expect(JSON.parse(orgRow.metadata).target).toBe('convertHandler'); + }); + + it('leaves archived rows alone — they are a record of what was, not served metadata', async () => { + const { engine, tables } = makeStubEngine([{ ...legacyObjectRow, state: 'archived' }]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.scanned).toBe(0); + expect(JSON.parse(metaRows(tables)[0]!.metadata).fields.amount.conditionalRequired).toBeDefined(); + }); + + it('restricts to the requested types', async () => { + const { engine } = makeStubEngine([legacyObjectRow, legacyActionRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true, types: ['action'] }); + + expect(report.scanned).toBe(1); + expect(report.rows[0]).toMatchObject({ type: 'action', outcome: 'rewritten' }); + }); +}); + +describe('migrateStoredMetadata — what it declines to touch, loudly (#4327)', () => { + it('skips flow rows and names the seam that owns them', async () => { + const legacyFlow = { + type: 'flow', + name: 'purge_flow', + metadata: { + name: 'purge_flow', + nodes: [{ id: 'n1', type: 'delete_record', config: { objectName: 'lead', filters: { status: 'stale' } } }], + }, + }; + const { engine, tables } = makeStubEngine([legacyFlow]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.skipped).toBe(1); + expect(report.rewritten).toBe(0); + expect(report.rows[0]!.reason).toMatch(/registerFlow/); + expect(JSON.parse(metaRows(tables)[0]!.metadata).nodes[0].config.filters).toEqual({ status: 'stale' }); + // A skipped row is a documented carve-out, not unfinished work: it does + // not fail the run's verdict, but the report always names it. + expect(storedMigrationClean(report)).toBe(true); + }); + + it('skips a type with no repository write path rather than rewriting it without history', async () => { + // `agent` is allowOrgOverride:false + allowRuntimeCreate:false, so + // `saveMetaItem` would take the legacy raw-engine branch: no history row + // and a forced `state: 'active'`. Declining beats a silent half-write. + const { engine, tables } = makeStubEngine([ + { type: 'agent', name: 'legacy_agent', metadata: { name: 'legacy_agent', label: 'Legacy' } }, + ]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.skipped).toBe(1); + expect(report.rows[0]!.reason).toMatch(/no repository write path/); + expect(historyRows(tables)).toHaveLength(0); + }); + + it('reports a row whose body is not JSON instead of throwing the whole run away', async () => { + const { engine } = makeStubEngine([ + { type: 'object', name: 'broken', metadata: '{ not json' }, + legacyObjectRow, + ]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.failed).toBe(1); + expect(report.rewritten).toBe(1); + expect(report.rows.find((r) => r.name === 'broken')!.reason).toMatch(/not valid JSON/); + expect(storedMigrationClean(report)).toBe(false); + }); + + it('reports — and does not write — a row that still fails the schema after conversion', async () => { + // `fields` as a number is a genuine contract violation no conversion + // owns. `saveMetaItem`'s 422 is correct, and the row keeps reading + // through the chain: this pass records the refusal rather than + // bypassing the gate that new rows are held to. + const { engine, tables } = makeStubEngine([ + { + type: 'object', + name: 'corrupt_thing', + metadata: { + name: 'corrupt_thing', + label: 'Corrupt', + fields: { amount: { type: 'currency', conditionalRequired: 'x' } }, + // an off-contract key the schema rejects and no conversion lowers + listViews: 42, + }, + }, + ]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.failed).toBe(1); + expect(report.rewritten).toBe(0); + expect(report.rows[0]!.reason).toMatch(/invalid_metadata/); + expect(historyRows(tables)).toHaveLength(0); + expect(JSON.parse(metaRows(tables)[0]!.metadata).fields.amount.conditionalRequired).toBe('x'); + }); + + it('does not clobber a row a concurrent writer moved — the optimistic lock is real', async () => { + const { engine, tables } = makeStubEngine([legacyObjectRow]); + // Someone else saved between the pass's scan and its write: the scan is + // handed the checksum the body was read under, the row on disk has + // already moved on. + metaRows(tables)[0]!.checksum = 'sha256:moved_by_someone_else'; + const protocol = new ObjectStackProtocolImplementation(engine); + const originalFind = engine.find.bind(engine); + let served = false; + engine.find = async (t: string, opts?: any) => { + const rows = await originalFind(t, opts); + if (t === 'sys_metadata' && !served) { + served = true; + return rows.map((r: any) => ({ ...r, checksum: 'sha256:stale' })); + } + return rows; + }; + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.failed).toBe(1); + expect(report.rows[0]!.reason).toMatch(/metadata_conflict/); + // The other writer's row is untouched. + expect(metaRows(tables)[0]!.checksum).toBe('sha256:moved_by_someone_else'); + }); +}); + +describe('formatStoredMigrationReport (#4327)', () => { + it('leads with the verdict when everything is already canonical', async () => { + const { engine } = makeStubEngine([canonicalObjectRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + const lines = formatStoredMigrationReport(await protocol.migrateStoredMetadata()); + expect(lines.join('\n')).toMatch(/already on protocol/); + }); + + it('refuses to call an empty scan clean — that is the wrong-directory reading too', async () => { + const { engine } = makeStubEngine([]); + const protocol = new ObjectStackProtocolImplementation(engine); + const report = await protocol.migrateStoredMetadata(); + expect(report.scanned).toBe(0); + const text = formatStoredMigrationReport(report).join('\n'); + expect(text).toMatch(/attests nothing/); + expect(text).not.toMatch(/already on protocol/); + }); + + it('prints each pending row with the conversion that would fire', async () => { + const { engine } = makeStubEngine([legacyObjectRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + const text = formatStoredMigrationReport(await protocol.migrateStoredMetadata()).join('\n'); + expect(text).toMatch(/object\/crm_invoice \[env-wide\]/); + expect(text).toMatch(/conditionalRequired → requiredWhen/); + }); +}); diff --git a/packages/metadata-protocol/src/protocol.ts b/packages/metadata-protocol/src/protocol.ts index c4e8a124c3..c43dd79b8a 100644 --- a/packages/metadata-protocol/src/protocol.ts +++ b/packages/metadata-protocol/src/protocol.ts @@ -27,10 +27,10 @@ import { type DroppedFieldsEvent, type QueryAST, } from '@objectstack/spec/data'; import { PLURAL_TO_SINGULAR, SINGULAR_TO_PLURAL } from '@objectstack/spec/shared'; -import { applyConversionsToStoredItem } from '@objectstack/spec'; +import { applyConversionsToStoredItem, type ConversionNotice } from '@objectstack/spec'; import { type FormView, isAggregatedViewContainer } from '@objectstack/spec/ui'; import { METADATA_FORM_REGISTRY, CORE_SERVICE_PROVIDER, serviceUnavailableMessage, inProcessServiceMessage } from '@objectstack/spec/system'; -import { DEFAULT_METADATA_TYPE_REGISTRY, getMetadataTypeSchema, getMetadataTypeActions, getMetadataCreateSeed } from '@objectstack/spec/kernel'; +import { DEFAULT_METADATA_TYPE_REGISTRY, getMetadataTypeSchema, getMetadataTypeActions, getMetadataCreateSeed, PROTOCOL_VERSION } from '@objectstack/spec/kernel'; import { extractProtection, evaluateLockForWrite, @@ -49,6 +49,11 @@ import { decorateMetadataItems, type MetadataDiagnostics, } from './metadata-diagnostics.js'; +import type { + StoredMigrationNotice, + StoredMigrationReport, + StoredMigrationRow, +} from './stored-migration.js'; /** * Canonical Zod schema per metadata type lives in @@ -1707,19 +1712,45 @@ export class ObjectStackProtocolImplementation implements */ private convertStoredItem(type: string, data: unknown): unknown { const singular = PLURAL_TO_SINGULAR[type] ?? type; - if (singular === 'flow') return data; - return applyConversionsToStoredItem(singular, data, { + return this.convertStoredItemDetailed(type, data, (n) => { + const name = (data as { name?: unknown } | null | undefined)?.name; + const key = `${n.conversionId}|${singular}|${String(name ?? '')}`; + if (this.storedConversionWarned.has(key)) return; + this.storedConversionWarned.add(key); + console.warn( + `[Protocol] stored ${singular}/${String(name ?? '')} carries a pre-protocol shape; ` + + `${n.message} The row itself is unchanged — re-save it (Studio edit → save, or run ` + + `"os migrate meta --stored --apply") to persist the canonical shape.`, + ); + }).item; + } + + /** + * {@link convertStoredItem} with the chain's notices handed back instead of + * only logged — what {@link migrateStoredMetadata} reports per row (#4327). + * + * The notices ARE the change signal: a conversion emits exactly one per + * rewrite it performs (ADR-0087 D2 "loud"), so an empty list means the row + * is already canonical and there is nothing to persist. Comparing bodies + * instead would be weaker — the pass is copy-on-write, so an untouched + * branch is shared and a re-serialized identical body can still differ in + * key order. + */ + private convertStoredItemDetailed( + type: string, + data: unknown, + onNotice?: (notice: ConversionNotice) => void, + ): { item: unknown; notices: ConversionNotice[] } { + const singular = PLURAL_TO_SINGULAR[type] ?? type; + if (singular === 'flow') return { item: data, notices: [] }; + const notices: ConversionNotice[] = []; + const item = applyConversionsToStoredItem(singular, data, { onNotice: (n) => { - const name = (data as { name?: unknown } | null | undefined)?.name; - const key = `${n.conversionId}|${singular}|${String(name ?? '')}`; - if (this.storedConversionWarned.has(key)) return; - this.storedConversionWarned.add(key); - console.warn( - `[Protocol] stored ${singular}/${String(name ?? '')} carries a pre-protocol shape; ` + - `${n.message} The row itself is unchanged — re-save it (Studio edit → save) to persist the canonical shape.`, - ); + notices.push(n); + onNotice?.(n); }, }); + return { item, notices }; } constructor( @@ -5858,10 +5889,19 @@ export class ObjectStackProtocolImplementation implements return true; } - async saveMetaItem(request: { type: string, name: string, item?: any, organizationId?: string, parentVersion?: string | null, actor?: string, force?: boolean, mode?: 'draft' | 'publish', packageId?: string | null }) { + async saveMetaItem(request: { type: string, name: string, item?: any, organizationId?: string, parentVersion?: string | null, actor?: string, force?: boolean, mode?: 'draft' | 'publish', packageId?: string | null, source?: string }) { if (!request.item) { throw new Error('Item data is required'); } + // What the history row, the audit row and the watch event record as the + // origin of this write. Defaults to this method — the ordinary Studio / + // REST / SDK save. The only caller that overrides it is + // {@link migrateStoredMetadata} (`'migrate-stored'`), so an operator + // reading a diff can tell a canonicalization pass from an author's edit + // (#4327). NOT request-derived: the REST layer builds this request field + // by field and never forwards a client-supplied `source`, so provenance + // stays something the server states, not something a caller claims. + const writeSource = request.source ?? 'protocol.saveMetaItem'; // Drop OUR OWN read decorations before anything reads the body (#4326). // The write path persists verbatim by design (ADR-0005 §Validation), so // the standard Studio round-trip — GET (decorated) → edit → PUT the whole @@ -5924,7 +5964,7 @@ export class ObjectStackProtocolImplementation implements ...(request.organizationId ? { organizationId: request.organizationId } : {}), operation: 'save', ...(request.actor ? { actor: request.actor } : {}), - source: 'protocol.saveMetaItem', + source: writeSource, }); if (lockErr) throw lockErr; } @@ -6182,7 +6222,7 @@ export class ObjectStackProtocolImplementation implements const result = await repo.put(ref, request.item, { parentVersion, actor: request.actor ?? 'system', - source: 'protocol.saveMetaItem', + source: writeSource, intent, state: mode === 'draft' ? 'draft' : 'active', ...(request.packageId !== undefined ? { packageId: request.packageId } : {}), @@ -6205,7 +6245,7 @@ export class ObjectStackProtocolImplementation implements outcome: 'allowed', code: 'ok', ...(request.actor ? { actor: request.actor } : {}), - source: 'protocol.saveMetaItem', + source: writeSource, note: mode === 'draft' ? 'draft' : 'active', }); // [ADR-0094] Awaited projection BEFORE the fire-and-forget @@ -6344,6 +6384,205 @@ export class ObjectStackProtocolImplementation implements } } + /** + * `os migrate meta --stored` — canonicalize `sys_metadata` rows in place so + * the read-path conversion chain has a finish line (#4327). + * + * #4317 made every stored-row rehydration seam replay the full ADR-0087 + * chain, so a row written under any past protocol *reads* canonical forever + * ({@link convertStoredItem}). The rows themselves stayed legacy: the chain + * re-lowers them on every load and each one emits a conversion notice per + * process. This pass ends that for a deployment that runs it — same chain, + * same policy, result written back — while the read path stays the + * guarantee for every deployment that does not (#3855: operator-run + * migrations cannot be relied upon, so nothing here is load-bearing). + * + * ## What it walks + * + * `active` and `draft` rows, every org (the env-wide `organization_id IS + * NULL` bucket included). `archived` / `deprecated` rows are deliberately + * not read: they are not served metadata, and rewriting them would edit a + * record of what *was*. `sys_metadata_history` is untouched for the same + * reason the addendum gives — converting a version body would break the + * checksum↔body pairing. + * + * ## How it writes + * + * Through {@link saveMetaItem}, not the repository directly, so a rewritten + * row gets everything an author's save gets: the schema gate, a + * `sys_metadata_history` row, a fresh checksum, the mutation projectors and + * the watch event Studio's HMR consumes. Three deliberate arguments: + * + * - `parentVersion: row.checksum` — a true optimistic lock. A concurrent + * writer that moved the row between our read and our write gets a 409, + * reported as `failed`, never a clobber. + * - `force: true` — the destructive-change diff compares the *stored* + * body (which `getMetaItem` already serves converted) against the body we + * are about to write (the same conversion). It is empty by construction, + * and there is no author here for a confirmation prompt to reach. + * - `source: 'migrate-stored'` — so a history diff distinguishes a + * canonicalization pass from an edit someone made. + * + * ## What it declines to touch, and says so + * + * - **`flow` rows.** Flow-node conversions carry ADR-0078's open-namespace + * conflict guard, which needs the automation engine's live executor + * registry; this layer does not have it, so flows canonicalize at + * `AutomationEngine.registerFlow` and are reported `skipped` here. + * - **Types with no repository write path** (neither `allowOrgOverride` nor + * `allowRuntimeCreate`). `saveMetaItem` routes those down the legacy + * raw-engine branch, which records no history and forces `state: + * 'active'` — a historyless rewrite that could also promote a draft is + * not what this pass promises, so it declines instead. + * - **Rows that still fail the current schema after conversion.** + * `saveMetaItem` rejects them (422) and that rejection is correct: the + * body is a genuine contract violation, not chain-owned history. They + * surface as `failed` with the validation message, keep reading through + * the chain, and stay fixable in Studio. + */ + async migrateStoredMetadata(request: { + /** Write. Omitted / false = preview: reports what it would do, writes nothing. */ + apply?: boolean; + /** Restrict to these metadata types (singular or plural spelling). */ + types?: string[]; + /** Recorded as the writer on the history + audit rows. */ + actor?: string; + } = {}): Promise { + const apply = request.apply === true; + const typeFilter = request.types && request.types.length > 0 + ? new Set(request.types.map((t) => PLURAL_TO_SINGULAR[t] ?? t)) + : null; + + const report: StoredMigrationReport = { + apply, + protocol: PROTOCOL_VERSION, + scanned: 0, + canonical: 0, + pending: 0, + rewritten: 0, + skipped: 0, + failed: 0, + rows: [], + }; + + // Two scoped queries rather than one unfiltered scan: `state` is an + // equality column and these are the only two states that are SERVED + // metadata. Archived bodies are never even read. + const rows: any[] = []; + for (const state of ['active', 'draft'] as const) { + rows.push(...await this.engine.find('sys_metadata', { where: { state } })); + } + + for (const row of rows) { + const rawType = String(row.type ?? ''); + const singular = PLURAL_TO_SINGULAR[rawType] ?? rawType; + if (typeFilter && !typeFilter.has(singular)) continue; + report.scanned++; + + const state: 'active' | 'draft' = row.state === 'draft' ? 'draft' : 'active'; + const organizationId: string | null = row.organization_id ?? null; + const packageId: string | null = row.package_id ?? null; + const base = { + id: String(row.id ?? ''), + type: singular, + name: String(row.name ?? ''), + organizationId, + packageId, + state, + notices: [] as StoredMigrationNotice[], + }; + // An already-canonical row is counted, never itemised: on a healthy + // deployment that is every row, and a report listing all of them + // would bury the handful that actually need something. + const record = (entry: StoredMigrationRow): void => { + if (entry.outcome === 'canonical') { + report.canonical++; + return; + } + report[entry.outcome]++; + report.rows.push(entry); + }; + + let body: unknown; + try { + body = typeof row.metadata === 'string' ? JSON.parse(row.metadata) : row.metadata; + } catch (e: any) { + record({ + ...base, + outcome: 'failed', + reason: `the stored body is not valid JSON (${e?.message ?? String(e)})`, + }); + continue; + } + + if (singular === 'flow') { + record({ + ...base, + outcome: 'skipped', + reason: 'flows canonicalize at AutomationEngine.registerFlow — the node-type ' + + 'conflict guard needs the live executor registry this layer does not have', + }); + continue; + } + + const overlayAllowed = ObjectStackProtocolImplementation.isOverlayAllowed(singular); + const runtimeCreateAllowed = ObjectStackProtocolImplementation.isRuntimeCreateAllowed(singular); + if (!overlayAllowed && !runtimeCreateAllowed) { + record({ + ...base, + outcome: 'skipped', + reason: `type '${singular}' has no repository write path (allowOrgOverride and ` + + 'allowRuntimeCreate are both false), so a rewrite would record no history', + }); + continue; + } + + const { item, notices } = this.convertStoredItemDetailed(singular, body); + if (notices.length === 0) { + record({ ...base, outcome: 'canonical' }); + continue; + } + const flattened: StoredMigrationNotice[] = notices.map((n) => ({ + conversionId: n.conversionId, + surface: n.surface, + from: n.from, + to: n.to, + path: n.path, + message: n.message, + })); + + if (!apply) { + record({ ...base, notices: flattened, outcome: 'pending' }); + continue; + } + + try { + await this.saveMetaItem({ + type: singular, + name: base.name, + item, + mode: state === 'draft' ? 'draft' : 'publish', + parentVersion: row.checksum ?? null, + packageId, + force: true, + source: 'migrate-stored', + actor: request.actor ?? 'migrate-stored', + ...(organizationId ? { organizationId } : {}), + }); + record({ ...base, notices: flattened, outcome: 'rewritten' }); + } catch (e: any) { + record({ + ...base, + notices: flattened, + outcome: 'failed', + reason: e?.message ?? String(e), + }); + } + } + + return report; + } + /** * Yield the durable change-log for a single metadata item — every * put/delete recorded in `sys_metadata_history` for `(org, type, name)`, diff --git a/packages/metadata-protocol/src/stored-migration.ts b/packages/metadata-protocol/src/stored-migration.ts new file mode 100644 index 0000000000..0cccec8a19 --- /dev/null +++ b/packages/metadata-protocol/src/stored-migration.ts @@ -0,0 +1,180 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The report shape + renderer for the stored-metadata canonicalization pass + * (#4327), the follow-up to #3903's read-path guarantee. + * + * #4317 closed the correctness gap from the read side: every stored-row + * rehydration seam replays the full ADR-0087 conversion chain + * (`applyConversionsToStoredItem`, retired entries included), so a row written + * under any past protocol is *served* canonical forever. What it deliberately + * did not do is make the rows themselves canonical — a pre-17 row keeps its + * legacy bytes, the chain re-lowers it on every load, and each affected row + * emits one conversion notice per process. + * + * {@link ObjectStackProtocolImplementation.migrateStoredMetadata} is the + * operator-run pass that ends that: same chain, same policy, but the result is + * written back through the normal write path so the row stops carrying the old + * dialect. This module owns the vocabulary that pass reports in, kept beside + * the types rather than in the CLI so an admin route renders the same run the + * same way. + * + * **Not load-bearing.** #3855's conclusion still holds — an operator-run + * migration cannot be relied on, so the read path stays the guarantee and + * nothing gates on this having run. No `sys_migration` flag is recorded for + * exactly that reason: a flag row would advertise a gate that does not exist. + * The verifiable statement operators wanted is the *re-run* — a second pass + * that reports every row canonical is the evidence, and it costs one command. + */ + +/** What the pass did with (or would do with) one `sys_metadata` row. */ +export type StoredMigrationOutcome = + /** The chain was a no-op — the row is already on protocol. Not itemised. */ + | 'canonical' + /** Preview: the chain would rewrite this row. Nothing was written. */ + | 'pending' + /** Apply: the canonical body was re-saved through the write path. */ + | 'rewritten' + /** Outside this pass's reach — see {@link StoredMigrationRow.reason}. */ + | 'skipped' + /** The row could not be read, or the re-save was refused. */ + | 'failed'; + +/** + * One conversion the chain applied to a row — the per-row detail a preview + * run prints, flattened from the spec's {@link ConversionNotice} to the + * fields an operator acts on. + */ +export interface StoredMigrationNotice { + /** The `MetadataConversion.id` that fired, e.g. `flow-node-crud-filter-alias`. */ + conversionId: string; + /** Dotted surface the conversion governs, e.g. `object.field.conditionalRequired`. */ + surface: string; + /** The off-spec token/shape found in the stored body. */ + from: string; + /** The canonical token/shape it was lowered to. */ + to: string; + /** Where in the item it applied, e.g. `objects[0].fields.amount`. */ + path: string; + /** The chain's own human-facing line. */ + message: string; +} + +/** Per-row result. Rows the chain left alone (`canonical`) are counted, not listed. */ +export interface StoredMigrationRow { + /** `sys_metadata.id` — the row this is about, so an operator can go look at it. */ + id: string; + /** Singular metadata type (`object`, `view`, …), normalized from the row's spelling. */ + type: string; + name: string; + /** `null` = the env-wide overlay bucket. */ + organizationId: string | null; + /** `null` = a package-less (global) overlay row. */ + packageId: string | null; + state: 'active' | 'draft'; + outcome: StoredMigrationOutcome; + /** The conversions this row carries. Empty unless the chain rewrote something. */ + notices: StoredMigrationNotice[]; + /** Why a `skipped` / `failed` row was not rewritten. Absent otherwise. */ + reason?: string; +} + +/** The whole run. `apply: false` is a preview — it writes nothing, by construction. */ +export interface StoredMigrationReport { + /** False = preview. A preview never writes, not even a row it would leave identical. */ + apply: boolean; + /** The protocol version the chain canonicalized *to* — what a clean run attests. */ + protocol: string; + /** Rows examined (after any `--type` filter; archived/deprecated rows are not read). */ + scanned: number; + /** Rows the chain left unchanged — already on protocol. */ + canonical: number; + /** Preview only: rows the chain would rewrite. Always 0 on an apply run. */ + pending: number; + /** Apply only: rows re-saved through the write path. */ + rewritten: number; + skipped: number; + failed: number; + /** Every row that is not `canonical`, in scan order. */ + rows: StoredMigrationRow[]; +} + +/** + * Is this deployment's stored metadata on protocol? + * + * True when nothing is left to convert and nothing was refused. `skipped` rows + * deliberately do NOT count against it: they name a seam that owns them + * (flows canonicalize at `AutomationEngine.registerFlow`) or a type this pass + * has no history-recording write path for — both are documented carve-outs, not + * work this command left half-done. They are still printed, so the operator + * sees what the verdict does not cover. + */ +export function storedMigrationClean(report: StoredMigrationReport): boolean { + return report.pending === 0 && report.failed === 0; +} + +/** Render a run for a terminal. One line per non-canonical row, notices nested. */ +export function formatStoredMigrationReport(report: StoredMigrationReport): string[] { + const lines: string[] = []; + lines.push( + `Examined ${report.scanned} stored metadata row(s) (active + draft, all orgs) ` + + `against the protocol ${report.protocol} conversion chain.`, + ); + + const converting = report.rows.filter((r) => r.outcome === 'pending' || r.outcome === 'rewritten'); + if (converting.length > 0) { + lines.push( + report.apply + ? `✓ Rewrote ${report.rewritten} row(s) carrying a pre-protocol shape:` + : `→ ${report.pending} row(s) carry a pre-protocol shape and would be rewritten:`, + ); + for (const row of converting) { + lines.push(` • ${row.type}/${row.name} ${describeScope(row)}`); + for (const n of row.notices) { + lines.push(` ${n.conversionId}: ${n.from} → ${n.to} at ${n.path}`); + } + } + } + + const skipped = report.rows.filter((r) => r.outcome === 'skipped'); + if (skipped.length > 0) { + lines.push(`⚠ ${skipped.length} row(s) are outside this pass — they keep reading through the chain:`); + for (const row of skipped) { + lines.push(` • ${row.type}/${row.name} ${describeScope(row)} — ${row.reason ?? 'skipped'}`); + } + } + + const failed = report.rows.filter((r) => r.outcome === 'failed'); + if (failed.length > 0) { + lines.push(`✗ ${failed.length} row(s) could not be rewritten:`); + for (const row of failed) { + lines.push(` • ${row.type}/${row.name} ${describeScope(row)} — ${row.reason ?? 'failed'}`); + } + } + + if (report.scanned === 0) { + // "Nothing to convert" and "nothing was looked at" are different claims, + // and only the first is a pass. A run pointed at the wrong project — the + // database line above names which one — would otherwise read as clean. + lines.push( + '⚠ No rows were examined, so this run attests nothing. An empty result is what ' + + 'a deployment that has never authored metadata looks like, and also what running ' + + 'from the wrong project root looks like — check the database named above.', + ); + } else if (converting.length === 0 && failed.length === 0) { + lines.push( + `✓ Every row examined is already on protocol ${report.protocol} — ` + + 'the read-path conversion pass is a no-op here.', + ); + } + return lines; +} + +/** `[org=… package=… draft]` — only the parts that are not the default. */ +function describeScope(row: StoredMigrationRow): string { + const parts: string[] = []; + parts.push(row.organizationId ? `org=${row.organizationId}` : 'env-wide'); + if (row.packageId) parts.push(`package=${row.packageId}`); + if (row.state === 'draft') parts.push('draft'); + return `[${parts.join(', ')}]`; +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 6c7903a29f..b7f6b44014 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -402,6 +402,9 @@ importers: '@objectstack/metadata': specifier: workspace:* version: link:../metadata + '@objectstack/metadata-protocol': + specifier: workspace:* + version: link:../metadata-protocol '@objectstack/objectql': specifier: workspace:^ version: link:../objectql