From 24d54fc11853d86059fac7b5d84683c0c6b1908f Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 08:08:32 +0000 Subject: [PATCH 1/3] feat(migrate,metadata-protocol): os migrate meta --stored rewrites sys_metadata rows in place (#4327) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit #4317 closed the correctness gap from the read side: every stored-row rehydration seam replays the full ADR-0087 conversion chain, retired entries included, so a row written under any past protocol is served canonical forever. The rows themselves stayed legacy — the chain re-lowers them on every load and each logs a conversion notice per process. Until now the only things that rewrote such a row were a Studio re-save and duplicatePackage. `os migrate meta --stored` walks sys_metadata (active + draft, all orgs), replays the same applyConversionsToStoredItem pass, and re-saves each changed body through saveMetaItem — so a rewritten row gets a sys_metadata_history entry, a fresh checksum and the mutation projectors, exactly like an author's save. The history row's source is `migrate-stored`, distinguishing an upgrade from an edit. parentVersion is the row's own checksum, so a concurrent writer produces a 409 the report names rather than a clobber. Preview is the default and --apply the only writing mode, matching its two siblings and #3617's "a dry run changes nothing"; an apply run refuses to start while another process holds the SQLite database. Nothing gates on this having run (#3855) and no sys_migration flag is recorded — a flag would advertise enforcement that does not exist. What a run buys is hygiene plus an assertable verdict: nothing left to do exits 0, work remaining exits 1. Three carve-outs are reported rather than counted as done: flow rows (their seam is AutomationEngine.registerFlow, which holds the executor registry the node-type conflict guard needs), types with no repository write path (agent), and rows that still fail the current schema after conversion. An empty scan says it attests nothing rather than reading as a pass. Also: protocol.migrateStoredMetadata() returns the same structured report an admin route would render, and saveMetaItem takes an optional `source` for its history/audit rows — server-stated, never request-derived. Closes #4327 Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WoZPKPDqJ7WB7z84xk9y3f --- .changeset/migrate-meta-stored-rewrite.md | 62 +++ content/docs/deployment/cli.mdx | 60 +++ ...0087-metadata-protocol-upgrade-contract.md | 35 ++ packages/cli/package.json | 1 + .../migrate/meta.stored-flags.test.ts | 68 +++ packages/cli/src/commands/migrate/meta.ts | 327 +++++++++++++- packages/metadata-protocol/src/index.ts | 8 + .../src/protocol.stored-migration.test.ts | 427 ++++++++++++++++++ packages/metadata-protocol/src/protocol.ts | 271 ++++++++++- .../metadata-protocol/src/stored-migration.ts | 180 ++++++++ pnpm-lock.yaml | 3 + 11 files changed, 1418 insertions(+), 24 deletions(-) create mode 100644 .changeset/migrate-meta-stored-rewrite.md create mode 100644 packages/cli/src/commands/migrate/meta.stored-flags.test.ts create mode 100644 packages/metadata-protocol/src/protocol.stored-migration.test.ts create mode 100644 packages/metadata-protocol/src/stored-migration.ts diff --git a/.changeset/migrate-meta-stored-rewrite.md b/.changeset/migrate-meta-stored-rewrite.md new file mode 100644 index 0000000000..3e90b8ec56 --- /dev/null +++ b/.changeset/migrate-meta-stored-rewrite.md @@ -0,0 +1,62 @@ +--- +"@objectstack/metadata-protocol": minor +"@objectstack/cli": patch +--- + +feat(migrate,metadata-protocol): `os migrate meta --stored` rewrites sys_metadata rows so the read-path chain has a finish line (#4327) + +#4317 closed the correctness gap from the read side: every stored-row +rehydration seam replays the full ADR-0087 conversion chain, retired entries +included, so a row written under any past protocol is *served* canonical +forever. What it deliberately did not do is make the rows themselves canonical. +A pre-17 row keeps its legacy bytes, the chain re-lowers it on every load, and +each affected row logs one conversion notice per process — deduped, but back +every boot. Until now the only things that ever rewrote such a row were a Studio +re-save and `duplicatePackage`. + +**`os migrate meta --stored`** is the pass that ends it for a deployment that +runs it. It walks `sys_metadata` — `active` and `draft`, every organization — +replays the same `applyConversionsToStoredItem` chain, and re-saves each changed +body through the normal write path, so a rewritten row gets a +`sys_metadata_history` entry, a fresh checksum and the mutation projectors, +exactly like an author's save. The history row's `source` is `migrate-stored`, +so a later diff distinguishes an upgrade from somebody's edit. + +```bash +os migrate meta --stored # preview: per-row report, writes nothing +os migrate meta --stored --apply # rewrite the rows (prompts) +os migrate meta --stored --apply --yes --json # CI / scripts +os migrate meta --stored --type view # restrict to a type (repeatable) +``` + +**Preview is the default and `--apply` is the only writing mode** — the house +rule its siblings already keep (#3617's "a dry run changes nothing"), and it +applies with more force here because what moves is metadata: every affected +row's checksum and a history entry per row. An apply run also refuses to start +while another process holds the SQLite database, for the same reason +`os migrate files-to-references --apply` does. + +**Nothing gates on this having run.** #3855's conclusion stands — an +operator-run migration cannot be relied upon, so the read path remains the +guarantee for every deployment, and no `sys_migration` flag is recorded (a flag +would advertise enforcement that does not exist). What a run buys is hygiene — +rows stop carrying pre-protocol dialects, so diffs, exports and history are +clean going forward, and the recurring notices go quiet — plus one thing that +was previously unobtainable: **an operator can assert it.** A run with nothing +left to do exits `0`, a deployment with rows still on an old dialect exits `1`, +so "my metadata is on protocol N" becomes a CI check rather than a belief. + +Three things the pass declines, and reports rather than counting as done: +`flow` rows (their seam is `AutomationEngine.registerFlow`, which holds the +executor registry the node-type conflict guard needs), types with no repository +write path (`agent` — rewriting there would record no history and force a draft +live), and rows that still fail the current schema after conversion (a genuine +contract violation the write path is right to refuse; it keeps reading through +the chain and stays fixable in Studio). + +Also new, and usable without the CLI: `protocol.migrateStoredMetadata()` returns +the same structured report an admin route would render, and `saveMetaItem` +accepts an optional `source` for the history/audit rows. `source` is not +request-derived — the REST layer builds its save request field by field and +never forwards a client-supplied value, so provenance stays something the server +states rather than something a caller claims. diff --git a/content/docs/deployment/cli.mdx b/content/docs/deployment/cli.mdx index 9f9be95817..bcdc815aeb 100644 --- a/content/docs/deployment/cli.mdx +++ b/content/docs/deployment/cli.mdx @@ -576,6 +576,7 @@ written. | `os migrate plan` | Warns and continues — a plan writes nothing either way | | `os migrate apply` | **Refuses** (exit 1, `error: database_busy` under `--json`). Stop the other process, or pass `--force` | | `os migrate files-to-references --apply` | **Refuses** likewise — it rewrites rows, so a concurrent writer is at least as dangerous | +| `os migrate meta --stored --apply` | **Refuses** likewise — it rewrites `sys_metadata` rows, and a live process saving metadata is exactly the collision | The check applies to SQLite only: Postgres and MySQL take their own server-side locks. Only same-user processes are visible without elevated privileges, and a @@ -629,6 +630,7 @@ where the data lives. |---------|-------------| | `os migrate files-to-references` | Convert legacy file-field values to `sys_file` references, verify the ownership ledger, and record the deployment's migration flag | | `os migrate value-shapes` | Scan stored reference and structured-JSON field values against the platform's value contract, and record the deployment's migration flag when clean | +| `os migrate meta --stored` | Replay the metadata conversion chain over this deployment's `sys_metadata` rows and rewrite the ones still carrying a pre-protocol shape. Hygiene, not a gate — nothing depends on it having run | ```bash os migrate files-to-references # Dry run: full report, writes nothing @@ -761,6 +763,64 @@ closed gate logs that it is enforcing, and an app that declares neither class of field says nothing at all. So a running deployment always tells you the state of its own data — which is the question `os migrate meta` cannot answer. +#### `os migrate meta --stored` + +The two commands above are about **application data**. This one is about the +**metadata itself**, at rest: the `sys_metadata` rows Studio and the runtime +authoring APIs write. + +Those rows already *read* correctly whatever protocol they were written under — +every rehydration seam replays the full conversion chain, so a body from an +older major is served in today's canonical shape and always will be. What the +rows do not do is *change*: they keep their original bytes, the chain re-lowers +them on every load, and each one logs a conversion notice once per boot. This +command ends that for the deployment that runs it. + +```bash +os migrate meta --stored # Preview: per-row report, writes nothing +os migrate meta --stored --apply # Rewrite the rows (prompts) +os migrate meta --stored --apply --yes --json # CI / scripts +os migrate meta --stored --type view --type object # Restrict to a type (repeatable) +``` + +It walks `active` and `draft` rows across every organization (archived rows are +a record of what *was* and are never read), replays the same chain the read path +does, and re-saves each changed body through the normal write path — so a +rewritten row gets a `sys_metadata_history` entry, a fresh checksum, and the +mutation projectors, exactly like an author's save. The history entry's source +is `migrate-stored`, so a later diff shows which changes were an upgrade and +which were somebody's edit. + +Three things it deliberately declines, and names in the report rather than +counting as done: + +| Not rewritten | Why | +| :--- | :--- | +| `flow` rows | Flow-node conversions carry a conflict guard that needs the automation engine's live executor registry; flows canonicalize at their own seam when the engine loads them | +| Types with no repository write path (`agent`) | Their write path records no history and would force a draft live — a half-write is worse than leaving the row to the read path | +| Rows that still fail the current schema after conversion | That is a genuine contract violation, not chain-owned history. The write path's rejection is correct; fix the row in Studio | + + +`--apply` is the only writing mode, and it rewrites **metadata** — each affected +row's checksum moves and each gets a history entry. Preview first. Like the +other row-rewriting migration, an apply run refuses to start while another +process holds the SQLite database (`--force` overrides). + + +**Nothing gates on this having run.** The read path is the guarantee, for every +deployment, whether or not anyone runs this — an operator-run migration is not +something the platform can depend on. What running it buys is hygiene (cleaner +diffs, exports and history from here on, and the recurring boot notices go +quiet) plus one thing that was previously unobtainable: **you can assert it.** +A run with nothing left to do exits `0`; a deployment with rows still carrying +an old dialect exits `1`. So "my metadata is on protocol N" becomes a check +rather than a belief. + +Note the division of labour with the default mode: `os migrate meta --from N` +rewrites an **author's source** and reads no database; `--stored` rewrites **one +deployment's rows** and reads no config. Same chain, opposite ends of the +contract — which is why the two modes are mutually exclusive. + ### Scaffolding | Command | Alias | Description | diff --git a/docs/adr/0087-metadata-protocol-upgrade-contract.md b/docs/adr/0087-metadata-protocol-upgrade-contract.md index d11fe813d2..973574f8cd 100644 --- a/docs/adr/0087-metadata-protocol-upgrade-contract.md +++ b/docs/adr/0087-metadata-protocol-upgrade-contract.md @@ -421,3 +421,38 @@ diverged (#3903). This addendum extends the contract to data at rest: at boot would unhook live tables and make the row unfixable in Studio (availability over purity for data at rest; the same verdict reaches Studio as `_diagnostics` on every read). + +## Addendum (2026-08-01) — the stored chain gets a finish line (#4327) + +The addendum above makes a legacy row read canonical *forever*, which is the +correctness guarantee — and, read literally, also a promise that the chain runs +on that row forever. `os migrate meta --stored` +(`ObjectStackProtocolImplementation.migrateStoredMetadata`) lets a deployment +end that for itself: it walks `sys_metadata` (active + draft, all orgs), replays +the same `applyConversionsToStoredItem` pass, and re-saves each changed body +through `saveMetaItem` with `source: 'migrate-stored'` — history row, checksum, +mutation projectors and all. Preview is the default; `--apply` is the only +writing mode. + +- **Not load-bearing, and no flag.** #3855's conclusion stands: an operator-run + migration cannot be relied on, so the read path — not this — remains the + guarantee, and nothing gates on it having run. Deliberately no `sys_migration` + row either: unlike ADR-0104's two gates, a flag here would advertise + enforcement that does not exist. The verifiable statement operators wanted is + the **re-run** — a second pass reporting every row canonical exits 0, so "my + metadata is on protocol N" is a check rather than a belief. +- **The write path's gate is not bypassed.** A body that still fails the current + schema after conversion is refused (422) and reported, exactly as the bullet + above describes for reads: it is a genuine contract violation, and the pass + has no more standing to persist it than an author does. It keeps reading + through the chain and stays fixable in Studio. +- **The version layer stays verbatim.** `sys_metadata_history` is appended to, + never rewritten. Canonicalizing a past version's body would break the + checksum↔body pairing this contract depends on — the migration is a new + commit, not a rewrite of history. +- **What the pass does not cover, it names.** Flows (their seam is + `AutomationEngine.registerFlow`, which holds the executor registry the + conflict guard needs) and types with no repository write path are reported as + `skipped` with the reason, never counted as done. Giving flows the same finish + line needs a canonicalization entry point on the automation engine and is + tracked separately. diff --git a/packages/cli/package.json b/packages/cli/package.json index f300cf4b54..253ad9351d 100644 --- a/packages/cli/package.json +++ b/packages/cli/package.json @@ -55,6 +55,7 @@ "@objectstack/lint": "workspace:*", "@objectstack/mcp": "workspace:*", "@objectstack/metadata": "workspace:*", + "@objectstack/metadata-protocol": "workspace:*", "@objectstack/objectql": "workspace:^", "@objectstack/observability": "workspace:^", "@objectstack/platform-objects": "workspace:*", diff --git a/packages/cli/src/commands/migrate/meta.stored-flags.test.ts b/packages/cli/src/commands/migrate/meta.stored-flags.test.ts new file mode 100644 index 0000000000..c5f860b8c7 --- /dev/null +++ b/packages/cli/src/commands/migrate/meta.stored-flags.test.ts @@ -0,0 +1,68 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #4327 — the flag surface where `os migrate meta`'s two modes meet. + * + * One command, two subjects: `--from N` replays the ADR-0087 chain over an + * **author's source** (a config file, no database in reach), `--stored` replays + * it over **one deployment's** `sys_metadata` rows (a database, no config in + * reach). Everything below pins a boundary that a plausible-looking oclif + * declaration gets wrong. + */ +import { describe, expect, it } from 'vitest'; +import MigrateMeta, { storedOnlyFlagsIn } from './meta.js'; + +describe('storedOnlyFlagsIn (#4327)', () => { + it('reports the stored-only flags the operator typed', () => { + expect(storedOnlyFlagsIn(['--from', '16', '--apply'])).toEqual(['apply']); + expect(storedOnlyFlagsIn(['--from=16', '--type=view', '--force'])).toEqual(['force', 'type']); + expect(storedOnlyFlagsIn(['--from', '16', '-y'])).toEqual(['yes']); + }); + + it('says nothing about a run that typed none of them', () => { + expect(storedOnlyFlagsIn(['--from', '16', '--step', '--json'])).toEqual([]); + expect(storedOnlyFlagsIn([])).toEqual([]); + }); + + it('never double-reports --yes given both spellings', () => { + expect(storedOnlyFlagsIn(['--stored', '--yes', '-y'])).toEqual(['yes']); + }); + + it('reads argv, not the environment — an exported OS_DATABASE_URL is not a typed flag', () => { + // The trap that made `dependsOn` unusable: oclif fills `--database-url` + // from `OS_DATABASE_URL`, so a merely-exported env var would have counted + // as "provided" and broken `os migrate meta --from N` for anyone who has + // one set. Provenance comes from argv precisely so it cannot. + expect(storedOnlyFlagsIn(['--from', '16'])).toEqual([]); + expect(storedOnlyFlagsIn(['--stored', '--database-url', 'sqlite://x.db'])).toEqual(['database-url']); + }); +}); + +describe('MigrateMeta flag declarations (#4327)', () => { + const flags = MigrateMeta.flags as Record; + + it('does not declare --from required — that would reject every --stored run', () => { + expect(flags.from.required).not.toBe(true); + }); + + it('makes the authored-chain flags exclusive with --stored', () => { + // `--from` names a major an author wrote against; a stored row carries its + // own history and gets the full chain regardless. Accepting both would + // imply the stored pass honours a range it does not have. + for (const name of ['from', 'to', 'step', 'out']) { + expect(flags[name].exclusive).toContain('stored'); + } + }); + + it('leaves the stored-only flags free of dependsOn', () => { + // Enforced by `storedOnlyFlagsIn` instead: see the env-var case above. + for (const name of ['apply', 'yes', 'force', 'type', 'database-url']) { + expect(flags[name].dependsOn).toBeUndefined(); + } + }); + + it('defaults --apply off — preview is the only mode a bare run can have', () => { + expect(flags.apply.default).toBe(false); + expect(flags.stored.default).toBe(false); + }); +}); diff --git a/packages/cli/src/commands/migrate/meta.ts b/packages/cli/src/commands/migrate/meta.ts index f9c0f81a3b..a5a4aaa17f 100644 --- a/packages/cli/src/commands/migrate/meta.ts +++ b/packages/cli/src/commands/migrate/meta.ts @@ -3,6 +3,7 @@ import { Args, Command, Flags } from '@oclif/core'; import { writeFileSync } from 'node:fs'; import { resolve } from 'node:path'; +import { createInterface } from 'node:readline'; import chalk from 'chalk'; import { ObjectStackDefinitionSchema, @@ -25,10 +26,45 @@ import { createTimer, emitJson, } from '../../utils/format.js'; +import { bootSchemaStack } from '../../utils/schema-migrate.js'; +import { buildDataMigrationPlugins } from '../../utils/data-migration-plugins.js'; +import { OCCUPANCY_HINT, probeMigrationTarget } from '../../utils/migrate-occupancy-gate.js'; +import { describeOccupancy } from '../../utils/sqlite-occupancy.js'; + +async function confirm(question: string): Promise { + if (!process.stdin.isTTY) return false; // non-interactive → require --yes + const rl = createInterface({ input: process.stdin, output: process.stdout }); + try { + const answer: string = await new Promise((res) => rl.question(question, res)); + return /^y(es)?$/i.test(answer.trim()); + } finally { + rl.close(); + } +} /** The protocol major that introduced the per-deployment value-shape gates. */ const VALUE_SHAPE_GATE_MAJOR = 17; +/** Flags that mean something only in `--stored` mode (#4327). */ +const STORED_ONLY_FLAGS = ['apply', 'yes', 'force', 'type', 'database-url'] as const; + +/** + * Which stored-only flags the operator actually TYPED. + * + * oclif's own `dependsOn` cannot answer this: a boolean with `default: false` + * and an `env`-backed string both read as "provided" to it, so declaring + * `dependsOn: ['stored']` on `--database-url` would make a merely-exported + * `OS_DATABASE_URL` break `os migrate meta --from N`. The raw argv is the only + * signal for intent, so the guard reads that. + */ +export function storedOnlyFlagsIn(argv: readonly string[]): string[] { + const typed = STORED_ONLY_FLAGS.filter((f) => + argv.some((a) => a === `--${f}` || a.startsWith(`--${f}=`)), + ) as string[]; + if (argv.includes('-y') && !typed.includes('yes')) typed.push('yes'); + return typed; +} + interface PendingDataMigration { /** `sys_migration` row id the run records. */ id: string; @@ -120,16 +156,35 @@ function printPendingDataMigrations(pending: PendingDataMigration[]): void { * unsafe and lossy); `--out` writes the canonicalized stack as a JSON snapshot * the agent can diff and adopt. `--step` prints a per-hop checkpoint so a failure * can be bisected to the exact major. + * + * ## `--stored`: the same chain, over data at rest (#4327) + * + * The default mode above has one subject — the **author's source**, read from a + * config file, with no database in reach. `--stored` has the other: the + * `sys_metadata` rows of **one deployment**. Same chain, same canonical target, + * opposite ends of the contract, which is why they share a command rather than + * splitting into two that would both be called "migrate the metadata". + * + * They are mutually exclusive for the same reason: `--from` describes a + * protocol major an author wrote against, and a stored row already carries its + * own history — the stored pass replays the full chain (retired entries + * included) because a row at rest has no author to ask. See + * {@link MigrateMeta.runStored}. */ export default class MigrateMeta extends Command { static override description = - 'Replay the metadata protocol migration chain from a past major to current (ADR-0087 D3).'; + 'Replay the metadata protocol migration chain from a past major to current (ADR-0087 D3). ' + + 'With --stored, replay it over this deployment\'s sys_metadata rows instead of an authored config.'; static override examples = [ '$ os migrate meta --from 10', '$ os migrate meta --from 10 --step', '$ os migrate meta --from 11 --to 12 --json', '$ os migrate meta --from 10 --out migrated.stack.json', + '$ os migrate meta --stored', + '$ os migrate meta --stored --apply', + '$ os migrate meta --stored --apply --yes --json', + '$ os migrate meta --stored --type view --type object', ]; static override args = { @@ -138,23 +193,97 @@ export default class MigrateMeta extends Command { static override flags = { from: Flags.integer({ - description: 'The protocol major the metadata was authored against.', - required: true, + description: 'The protocol major the metadata was authored against (required without --stored).', + exclusive: ['stored'], }), to: Flags.integer({ description: `Target protocol major (defaults to this runtime's, ${PROTOCOL_MAJOR}).`, + exclusive: ['stored'], }), step: Flags.boolean({ description: 'Print a per-hop checkpoint (for per-major verify / bisection).', default: false, + exclusive: ['stored'], + }), + out: Flags.string({ + description: 'Write the migrated stack as a JSON snapshot to this path.', + exclusive: ['stored'], + }), + stored: Flags.boolean({ + description: + "Canonicalize this deployment's sys_metadata rows in place instead of an authored config " + + '(read-only preview unless --apply).', + default: false, + }), + 'database-url': Flags.string({ + description: '--stored: database to canonicalize (defaults to $OS_DATABASE_URL / the project DB)', + env: 'OS_DATABASE_URL', + }), + apply: Flags.boolean({ + description: '--stored: rewrite the rows (default is a read-only preview)', + default: false, + }), + yes: Flags.boolean({ + char: 'y', + description: '--stored: skip the --apply confirmation prompt', + default: false, + }), + force: Flags.boolean({ + description: '--stored: apply even when another process is using the database (SQLite occupancy check)', + default: false, + }), + type: Flags.string({ + description: '--stored: restrict to this metadata type (repeatable; default: every type)', + multiple: true, }), - out: Flags.string({ description: 'Write the migrated stack as a JSON snapshot to this path.' }), json: Flags.boolean({ description: 'Output the machine-readable migration result as JSON.' }), }; async run(): Promise { const { args, flags } = await this.parse(MigrateMeta); const timer = createTimer(); + + if (flags.stored) { + await this.runStored(flags, timer); + return; + } + + // A stored-only flag typed without `--stored` is refused rather than + // ignored: `--apply` in particular reads as "and write it", and the + // authored-source mode has nothing to write to. + const typed = storedOnlyFlagsIn(this.argv); + if (typed.length > 0) { + const message = + `${typed.map((f) => `--${f}`).join(', ')} only appl${typed.length > 1 ? 'y' : 'ies'} to ` + + '`os migrate meta --stored` (the pass over a deployment\'s sys_metadata rows). ' + + 'The authored-source chain reads a config file and writes nothing but --out.'; + if (flags.json) { + await emitJson({ error: 'stored_only_flag', flags: typed, message }, 0, { compact: true }); + this.exit(1); + return; + } + printError(message); + this.exit(1); + return; + } + + // `--from` is required for the authored-source chain and meaningless for + // `--stored` (a row carries its own history), so it is validated here + // rather than declared `required` — oclif would reject `--stored` runs. + if (flags.from === undefined) { + const message = + 'Missing required flag --from (the protocol major your metadata was authored against). ' + + 'To canonicalize a deployment\'s stored rows instead, run `os migrate meta --stored`.'; + if (flags.json) { + await emitJson({ error: 'missing_from_major', message }, 0, { compact: true }); + this.exit(1); + return; + } + printError(message); + this.exit(1); + return; + } + const fromMajor = flags.from; const toMajor = flags.to ?? PROTOCOL_MAJOR; if (!flags.json) printHeader('Migrate · meta'); @@ -169,13 +298,13 @@ export default class MigrateMeta extends Command { // D2 pass. Running the D2 pass here would leave the chain's diff empty. const normalized = normalizeStackInput(config as Record, { convert: false }); - if (!flags.json) printStep(`Replaying chain: protocol ${flags.from} → ${toMajor}…`); - const result = applyMetaMigrations(normalized, flags.from, toMajor); + if (!flags.json) printStep(`Replaying chain: protocol ${fromMajor} → ${toMajor}…`); + const result = applyMetaMigrations(normalized, fromMajor, toMajor); // Prove the migrated stack is schema-valid — the "generated, provably valid // diff" the consumer agent reviews (ADR-0087 D3/D5). const parsed = ObjectStackDefinitionSchema.safeParse(result.stack); - const specChanges = composeSpecChanges(flags.from, toMajor); + const specChanges = composeSpecChanges(fromMajor, toMajor); const dataMigrations = pendingDataMigrations(result.stack, result.fromMajor, result.toMajor); if (flags.json) { @@ -205,7 +334,7 @@ export default class MigrateMeta extends Command { } printInfo(`Config: ${chalk.white(absolutePath)}`); - printInfo(`Chain: protocol ${flags.from} → ${toMajor} (runtime ${PROTOCOL_VERSION})`); + printInfo(`Chain: protocol ${fromMajor} → ${toMajor} (runtime ${PROTOCOL_VERSION})`); console.log(''); if (result.applied.length === 0 && result.todos.length === 0) { @@ -281,4 +410,186 @@ export default class MigrateMeta extends Command { this.exit(1); } } + + /** + * `os migrate meta --stored` — canonicalize this deployment's `sys_metadata` + * rows so the read-path conversion chain has a finish line (#4327). + * + * #4317 made every stored-row rehydration seam replay the full ADR-0087 chain, + * so a row written under any past protocol *reads* canonical forever. The rows + * stayed legacy, though: the chain re-lowers them on every load and each one + * warns once per boot. This run ends that — same chain, same policy, but the + * result is written back through the normal write path (history row, checksum, + * mutation projectors) with `source: 'migrate-stored'`. + * + * **Preview by default; `--apply` is the only writing mode.** That is the + * house rule its two siblings already keep (`os migrate value-shapes`, + * `os migrate files-to-references`, #3617's "a dry run changes nothing"), and + * the reason applies with more force here: this rewrites *metadata*, so a + * surprise run would move every affected row's checksum and mint a history + * entry against each. + * + * Nothing gates on this having run — #3855's conclusion that operator-run + * migrations cannot be relied on still holds, and the read path stays the + * guarantee. What a run buys is hygiene plus one thing that was previously + * unobtainable: **an operator can now assert it.** A second pass reporting + * every row canonical exits 0; a deployment with work left exits 1, so "my + * metadata is on protocol N" becomes a CI check instead of a belief. + */ + private async runStored( + flags: { + json: boolean; + apply: boolean; + yes: boolean; + force: boolean; + type?: string[]; + 'database-url'?: string; + }, + timer: { elapsed: () => number; display: () => string }, + ): Promise { + const apply = flags.apply; + if (!flags.json) printHeader('Migrate · meta --stored'); + + // Occupancy gate — an apply run rewrites rows, so a live writer on the same + // SQLite file is the same hazard `os migrate files-to-references --apply` + // refuses for. Probed BEFORE boot (afterwards our own pool is what the probe + // finds) and before the prompt, so nobody confirms something we then refuse. + const occupancy = await probeMigrationTarget(flags['database-url']); + if (occupancy.status === 'busy' && apply && !flags.force) { + if (flags.json) { + await emitJson({ + error: 'database_busy', + database: occupancy.filename, + signal: occupancy.signal, + detail: occupancy.detail, + hint: OCCUPANCY_HINT, + }, 0, { compact: true }); + this.exit(1); + return; + } + printError(describeOccupancy(occupancy)); + printWarning(OCCUPANCY_HINT); + this.exit(1); + return; + } + if (occupancy.status === 'busy' && !flags.json) { + printWarning(apply + ? `--force: ${describeOccupancy(occupancy)} Rewriting anyway — the live process may save metadata mid-run.` + : `${describeOccupancy(occupancy)} The preview below writes nothing, but a live process may ` + + 'be saving metadata while it runs.'); + } + if (occupancy.status === 'unknown' && !flags.json) { + printWarning(`Could not check whether the database is in use — ${occupancy.detail}`); + } + + if (apply && !flags.yes) { + if (flags.json || !process.stdin.isTTY) { + if (flags.json) { + await emitJson({ error: 'confirmation_required', hint: 'pass --yes' }, 0, { compact: true }); + this.exit(1); + return; + } + printWarning( + 'Apply mode rewrites sys_metadata rows in place — each rewritten row gets a new checksum ' + + 'and a history entry. Re-run with --yes to confirm, or run without --apply to preview.', + ); + this.exit(1); + return; + } + const ok = await confirm( + chalk.bold('\nRewrite the stored metadata rows that carry a pre-protocol shape? [y/N] '), + ); + if (!ok) { + printInfo('Aborted — nothing written.'); + return; + } + } + + if (!flags.json) { + printStep(apply ? 'Booting data stack (APPLY mode)…' : 'Booting data stack (preview only)…'); + } + + let stack; + try { + // `PlatformObjectsPlugin` only — this pass needs `sys_metadata` and its + // history/audit siblings, which the protocol assembly registers itself. + // No storage adapter: unlike the file migration, nothing here reads bytes. + stack = await bootSchemaStack({ + ...(flags['database-url'] ? { databaseUrl: flags['database-url'] } : {}), + extraPlugins: await buildDataMigrationPlugins(), + }); + } catch (error: any) { + if (flags.json) { await emitJson({ error: error.message }, 0, { compact: true }); this.exit(1); return; } + printError(error.message || String(error)); + this.exit(1); + return; + } + + // Collected rather than thrown: `this.exit()` raises an oclif ExitError, + // which the catch below would then report as a bare "EEXIT: 1" over the + // real message. Decide the code here, exit after the stack is down. + let exitCode = 0; + try { + const protocol: any = stack.kernel.getService('protocol'); + if (typeof protocol?.migrateStoredMetadata !== 'function') { + throw new Error( + 'No metadata protocol on this stack — cannot walk sys_metadata. ' + + 'Run this from a project root whose stack registers the ObjectQL engine.', + ); + } + + const { formatStoredMigrationReport, storedMigrationClean } = + await import('@objectstack/metadata-protocol'); + + const report = await protocol.migrateStoredMetadata({ + apply, + ...(flags.type && flags.type.length > 0 ? { types: flags.type } : {}), + actor: 'os migrate meta --stored', + }); + const clean = storedMigrationClean(report); + if (!clean) exitCode = 1; + + if (flags.json) { + await emitJson({ database: stack.dbLabel, ...report, clean, duration: timer.elapsed() }); + } else { + printInfo(`Database: ${chalk.white(stack.dbLabel)}`); + console.log(''); + console.log(formatStoredMigrationReport(report).join('\n')); + console.log(''); + + if (report.scanned === 0) { + // Exits 0 — nothing is wrong — but does not claim a clean bill for a + // database it never read a row from. + printInfo(`No stored metadata to examine ${chalk.dim(`(${timer.display()})`)}`); + } else if (clean && apply) { + printSuccess( + `Stored metadata is on protocol ${report.protocol} — rewrote ${report.rewritten} row(s) ` + + `${chalk.dim(`(${timer.display()})`)}`, + ); + } else if (clean) { + printSuccess( + `Stored metadata is already on protocol ${report.protocol} — nothing to rewrite ` + + `${chalk.dim(`(${timer.display()})`)}`, + ); + } else if (report.failed > 0) { + printError( + `${report.failed} row(s) could not be rewritten. They keep reading canonically through the ` + + 'chain, so nothing is broken — but their stored bytes stay legacy until the reason above is fixed.', + ); + } else { + printWarning( + `${report.pending} row(s) carry a pre-protocol shape. They read canonically today (the chain ` + + 'runs on every load); re-run with --apply to persist it.', + ); + } + } + } catch (error: any) { + exitCode = 1; + if (flags.json) await emitJson({ error: error.message }, 0, { compact: true }); + else printError(error.message || String(error)); + } finally { + await stack.shutdown(); + } + if (exitCode !== 0) this.exit(exitCode); + } } diff --git a/packages/metadata-protocol/src/index.ts b/packages/metadata-protocol/src/index.ts index e7e28d01bb..303677250a 100644 --- a/packages/metadata-protocol/src/index.ts +++ b/packages/metadata-protocol/src/index.ts @@ -15,6 +15,14 @@ export type { ExtendedOperation, } from './sys-metadata-repository.js'; +export { formatStoredMigrationReport, storedMigrationClean } from './stored-migration.js'; +export type { + StoredMigrationNotice, + StoredMigrationOutcome, + StoredMigrationReport, + StoredMigrationRow, +} from './stored-migration.js'; + export { computeMetadataDiagnostics, computeViewReferenceDiagnostics, diff --git a/packages/metadata-protocol/src/protocol.stored-migration.test.ts b/packages/metadata-protocol/src/protocol.stored-migration.test.ts new file mode 100644 index 0000000000..8155d82d9c --- /dev/null +++ b/packages/metadata-protocol/src/protocol.stored-migration.test.ts @@ -0,0 +1,427 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * #4327 — `os migrate meta --stored`: the read-path conversion chain gets a + * finish line. + * + * #3903/#4317 made every stored-row rehydration seam replay the full chain, so + * a legacy row *reads* canonical forever. It stayed legacy on disk, though — + * re-lowered on every load, warning once per boot. `migrateStoredMetadata` + * writes the canonical body back through the normal write path so the row + * itself stops carrying the old dialect. + * + * Rows are seeded straight into the stub engine, deliberately bypassing + * `saveMetaItem`'s schema gate — exactly like a real row written years ago + * under an older protocol. What the pass then does with them is the contract + * under test: preview writes nothing, apply rewrites through the repository + * (history row + fresh checksum + `source: 'migrate-stored'`), and everything + * it declines to touch says so instead of being silently counted clean. + */ +import { describe, expect, it } from 'vitest'; +import { ObjectStackProtocolImplementation } from './protocol.js'; +import { formatStoredMigrationReport, storedMigrationClean } from './stored-migration.js'; + +interface Row { + id: string; + type: string; + name: string; + organization_id: string | null; + package_id: string | null; + state: string; + checksum: string | null; + metadata: string; +} + +function matches(r: Record, where: Record): boolean { + for (const [k, v] of Object.entries(where)) { + if (v === undefined) continue; + if ((r[k] ?? null) !== v) return false; + } + return true; +} + +/** + * A multi-table stub engine: `sys_metadata` seeded from `seedRows`, every other + * table (`sys_metadata_history`, `sys_metadata_audit`) created on first write. + * The repository write path needs all of them, and the history table is where + * this feature's central claim — "re-saved through the normal write path" — is + * actually observable. + */ +function makeStubEngine( + seedRows: Array & { type: string; name: string; metadata: unknown }>, +) { + let nextId = 0; + const tables = new Map[]>(); + tables.set( + 'sys_metadata', + seedRows.map((r) => ({ + id: `r_${++nextId}`, + organization_id: null, + package_id: null, + state: 'active', + checksum: `sha256:seed_${nextId}`, + ...r, + metadata: typeof r.metadata === 'string' ? r.metadata : JSON.stringify(r.metadata), + })), + ); + const rowsOf = (t: string): Record[] => { + let rows = tables.get(t); + if (!rows) tables.set(t, (rows = [])); + return rows; + }; + + const engine: any = { + async find(t: string, opts?: { where?: Record }) { + return rowsOf(t).filter((r) => matches(r, opts?.where ?? {})); + }, + async findOne(t: string, opts?: { where?: Record }) { + return rowsOf(t).find((r) => matches(r, opts?.where ?? {})) ?? null; + }, + async insert(t: string, row: Record) { + const withId = { id: row.id ?? `r_${++nextId}`, ...row }; + rowsOf(t).push(withId); + return withId; + }, + async update(t: string, patch: Record, opts: { where: Record }) { + const target = rowsOf(t).find((r) => matches(r, opts.where)); + if (target) Object.assign(target, patch); + return target ?? { id: 'x' }; + }, + async delete() { return { deleted: 0 }; }, + registry: { + listItems: () => [], + isPackageDisabled: () => false, + registerItem: () => { /* no-op */ }, + registerObject: () => { /* no-op */ }, + }, + }; + return { engine, tables }; +} + +const metaRows = (tables: Map[]>) => tables.get('sys_metadata')!; +const historyRows = (tables: Map[]>) => + tables.get('sys_metadata_history') ?? []; + +/** + * A protocol-≤16 object row: `conditionalRequired` was removed from the spec in + * 17 (#3855) and its conversion is `retiredFromLoadPath` — the authored load + * seam refuses it, the stored seam keeps lowering it, and this pass persists + * the lowering. + */ +const legacyObjectRow = { + type: 'object', + name: 'crm_invoice', + metadata: { + name: 'crm_invoice', + label: 'Invoice', + fields: { + status: { type: 'select', label: 'Status' }, + amount: { type: 'currency', label: 'Amount', conditionalRequired: "record.status == 'sent'" }, + }, + }, +}; + +/** The same object, already canonical — the shape every row ends up in. */ +const canonicalObjectRow = { + type: 'object', + name: 'crm_quote', + metadata: { + name: 'crm_quote', + label: 'Quote', + fields: { + status: { type: 'select', label: 'Status' }, + amount: { type: 'currency', label: 'Amount', requiredWhen: "record.status == 'sent'" }, + }, + }, +}; + +/** A pre-17 standalone action row still carrying the removed `execute` alias. */ +const legacyActionRow = { + type: 'action', + name: 'convert', + metadata: { name: 'convert', label: 'Convert', type: 'script', object: 'crm_invoice', execute: 'convertHandler' }, +}; + +describe('migrateStoredMetadata — preview (#4327)', () => { + it('reports the rows carrying a pre-protocol shape and writes nothing', async () => { + const { engine, tables } = makeStubEngine([legacyObjectRow, canonicalObjectRow]); + const before = JSON.stringify(metaRows(tables)); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata(); + + expect(report.apply).toBe(false); + expect(report.scanned).toBe(2); + expect(report.pending).toBe(1); + expect(report.canonical).toBe(1); + expect(report.rewritten).toBe(0); + // The whole point of a preview: the bytes are exactly as they were, and + // no history row was appended either. + expect(JSON.stringify(metaRows(tables))).toBe(before); + expect(historyRows(tables)).toHaveLength(0); + }); + + it('names the conversion per row, so a preview is actionable rather than a count', async () => { + const { engine } = makeStubEngine([legacyObjectRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata(); + + expect(report.rows).toHaveLength(1); + const row = report.rows[0]!; + expect(row).toMatchObject({ type: 'object', name: 'crm_invoice', outcome: 'pending', state: 'active' }); + expect(row.notices.length).toBeGreaterThan(0); + expect(row.notices[0]!.from).toBe('conditionalRequired'); + expect(row.notices[0]!.to).toBe('requiredWhen'); + // An already-canonical row is counted, never itemised — otherwise a + // healthy deployment's report is a wall of rows that need nothing. + expect(report.rows.every((r) => r.outcome !== 'canonical')).toBe(true); + }); + + it('is not "clean" while work remains — that verdict is what a CI gate reads', async () => { + const { engine } = makeStubEngine([legacyObjectRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + expect(storedMigrationClean(await protocol.migrateStoredMetadata())).toBe(false); + }); +}); + +describe('migrateStoredMetadata — apply (#4327)', () => { + it('rewrites the row in place with the canonical body', async () => { + const { engine, tables } = makeStubEngine([legacyObjectRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.rewritten).toBe(1); + expect(report.pending).toBe(0); + expect(storedMigrationClean(report)).toBe(true); + + const stored = JSON.parse(metaRows(tables)[0]!.metadata); + expect(stored.fields.amount.requiredWhen).toBe("record.status == 'sent'"); + expect('conditionalRequired' in stored.fields.amount).toBe(false); + }); + + it('goes through the normal write path — history row, fresh checksum, migrate-stored source', async () => { + const { engine, tables } = makeStubEngine([legacyObjectRow]); + const seededChecksum = metaRows(tables)[0]!.checksum; + const protocol = new ObjectStackProtocolImplementation(engine); + + await protocol.migrateStoredMetadata({ apply: true }); + + const history = historyRows(tables); + expect(history).toHaveLength(1); + expect(history[0]).toMatchObject({ + type: 'object', + name: 'crm_invoice', + source: 'migrate-stored', + // The lineage is intact: the new version's parent is the row's + // pre-migration checksum, not a null "created from nothing". + previous_checksum: seededChecksum, + }); + // A rewritten body is a new content hash — the row is no longer + // addressed by the checksum its legacy bytes had. + expect(metaRows(tables)[0]!.checksum).not.toBe(seededChecksum); + expect(metaRows(tables)[0]!.checksum).toBe(history[0]!.checksum); + }); + + it('re-running is a no-op — the second pass finds every row canonical', async () => { + const { engine, tables } = makeStubEngine([legacyObjectRow, legacyActionRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const first = await protocol.migrateStoredMetadata({ apply: true }); + expect(first.rewritten).toBe(2); + + const second = await protocol.migrateStoredMetadata({ apply: true }); + expect(second.scanned).toBe(2); + expect(second.canonical).toBe(2); + expect(second.rewritten).toBe(0); + expect(second.rows).toHaveLength(0); + // Idempotence is the operator's verifiable statement: nothing new was + // written on the second run either. + expect(historyRows(tables)).toHaveLength(2); + }); + + it('rewrites a DRAFT row as a draft — the pass never promotes staged work live', async () => { + const { engine, tables } = makeStubEngine([{ ...legacyObjectRow, state: 'draft' }]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.rewritten).toBe(1); + expect(report.rows[0]).toMatchObject({ state: 'draft', outcome: 'rewritten' }); + expect(metaRows(tables)[0]!.state).toBe('draft'); + }); + + it('walks every org, not just the env-wide bucket', async () => { + const { engine, tables } = makeStubEngine([ + legacyObjectRow, + { ...legacyActionRow, organization_id: 'org_a' }, + ]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.scanned).toBe(2); + expect(report.rewritten).toBe(2); + const orgRow = metaRows(tables).find((r) => r.organization_id === 'org_a')!; + expect(JSON.parse(orgRow.metadata).target).toBe('convertHandler'); + }); + + it('leaves archived rows alone — they are a record of what was, not served metadata', async () => { + const { engine, tables } = makeStubEngine([{ ...legacyObjectRow, state: 'archived' }]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.scanned).toBe(0); + expect(JSON.parse(metaRows(tables)[0]!.metadata).fields.amount.conditionalRequired).toBeDefined(); + }); + + it('restricts to the requested types', async () => { + const { engine } = makeStubEngine([legacyObjectRow, legacyActionRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true, types: ['action'] }); + + expect(report.scanned).toBe(1); + expect(report.rows[0]).toMatchObject({ type: 'action', outcome: 'rewritten' }); + }); +}); + +describe('migrateStoredMetadata — what it declines to touch, loudly (#4327)', () => { + it('skips flow rows and names the seam that owns them', async () => { + const legacyFlow = { + type: 'flow', + name: 'purge_flow', + metadata: { + name: 'purge_flow', + nodes: [{ id: 'n1', type: 'delete_record', config: { objectName: 'lead', filters: { status: 'stale' } } }], + }, + }; + const { engine, tables } = makeStubEngine([legacyFlow]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.skipped).toBe(1); + expect(report.rewritten).toBe(0); + expect(report.rows[0]!.reason).toMatch(/registerFlow/); + expect(JSON.parse(metaRows(tables)[0]!.metadata).nodes[0].config.filters).toEqual({ status: 'stale' }); + // A skipped row is a documented carve-out, not unfinished work: it does + // not fail the run's verdict, but the report always names it. + expect(storedMigrationClean(report)).toBe(true); + }); + + it('skips a type with no repository write path rather than rewriting it without history', async () => { + // `agent` is allowOrgOverride:false + allowRuntimeCreate:false, so + // `saveMetaItem` would take the legacy raw-engine branch: no history row + // and a forced `state: 'active'`. Declining beats a silent half-write. + const { engine, tables } = makeStubEngine([ + { type: 'agent', name: 'legacy_agent', metadata: { name: 'legacy_agent', label: 'Legacy' } }, + ]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.skipped).toBe(1); + expect(report.rows[0]!.reason).toMatch(/no repository write path/); + expect(historyRows(tables)).toHaveLength(0); + }); + + it('reports a row whose body is not JSON instead of throwing the whole run away', async () => { + const { engine } = makeStubEngine([ + { type: 'object', name: 'broken', metadata: '{ not json' }, + legacyObjectRow, + ]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.failed).toBe(1); + expect(report.rewritten).toBe(1); + expect(report.rows.find((r) => r.name === 'broken')!.reason).toMatch(/not valid JSON/); + expect(storedMigrationClean(report)).toBe(false); + }); + + it('reports — and does not write — a row that still fails the schema after conversion', async () => { + // `fields` as a number is a genuine contract violation no conversion + // owns. `saveMetaItem`'s 422 is correct, and the row keeps reading + // through the chain: this pass records the refusal rather than + // bypassing the gate that new rows are held to. + const { engine, tables } = makeStubEngine([ + { + type: 'object', + name: 'corrupt_thing', + metadata: { + name: 'corrupt_thing', + label: 'Corrupt', + fields: { amount: { type: 'currency', conditionalRequired: 'x' } }, + // an off-contract key the schema rejects and no conversion lowers + listViews: 42, + }, + }, + ]); + const protocol = new ObjectStackProtocolImplementation(engine); + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.failed).toBe(1); + expect(report.rewritten).toBe(0); + expect(report.rows[0]!.reason).toMatch(/invalid_metadata/); + expect(historyRows(tables)).toHaveLength(0); + expect(JSON.parse(metaRows(tables)[0]!.metadata).fields.amount.conditionalRequired).toBe('x'); + }); + + it('does not clobber a row a concurrent writer moved — the optimistic lock is real', async () => { + const { engine, tables } = makeStubEngine([legacyObjectRow]); + // Someone else saved between the pass's scan and its write: the scan is + // handed the checksum the body was read under, the row on disk has + // already moved on. + metaRows(tables)[0]!.checksum = 'sha256:moved_by_someone_else'; + const protocol = new ObjectStackProtocolImplementation(engine); + const originalFind = engine.find.bind(engine); + let served = false; + engine.find = async (t: string, opts?: any) => { + const rows = await originalFind(t, opts); + if (t === 'sys_metadata' && !served) { + served = true; + return rows.map((r: any) => ({ ...r, checksum: 'sha256:stale' })); + } + return rows; + }; + + const report = await protocol.migrateStoredMetadata({ apply: true }); + + expect(report.failed).toBe(1); + expect(report.rows[0]!.reason).toMatch(/metadata_conflict/); + // The other writer's row is untouched. + expect(metaRows(tables)[0]!.checksum).toBe('sha256:moved_by_someone_else'); + }); +}); + +describe('formatStoredMigrationReport (#4327)', () => { + it('leads with the verdict when everything is already canonical', async () => { + const { engine } = makeStubEngine([canonicalObjectRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + const lines = formatStoredMigrationReport(await protocol.migrateStoredMetadata()); + expect(lines.join('\n')).toMatch(/already on protocol/); + }); + + it('refuses to call an empty scan clean — that is the wrong-directory reading too', async () => { + const { engine } = makeStubEngine([]); + const protocol = new ObjectStackProtocolImplementation(engine); + const report = await protocol.migrateStoredMetadata(); + expect(report.scanned).toBe(0); + const text = formatStoredMigrationReport(report).join('\n'); + expect(text).toMatch(/attests nothing/); + expect(text).not.toMatch(/already on protocol/); + }); + + it('prints each pending row with the conversion that would fire', async () => { + const { engine } = makeStubEngine([legacyObjectRow]); + const protocol = new ObjectStackProtocolImplementation(engine); + const text = formatStoredMigrationReport(await protocol.migrateStoredMetadata()).join('\n'); + expect(text).toMatch(/object\/crm_invoice \[env-wide\]/); + expect(text).toMatch(/conditionalRequired → requiredWhen/); + }); +}); diff --git a/packages/metadata-protocol/src/protocol.ts b/packages/metadata-protocol/src/protocol.ts index cddb8e7cc1..95aa8e532e 100644 --- a/packages/metadata-protocol/src/protocol.ts +++ b/packages/metadata-protocol/src/protocol.ts @@ -27,10 +27,10 @@ import { type DroppedFieldsEvent, type QueryAST, } from '@objectstack/spec/data'; import { PLURAL_TO_SINGULAR, SINGULAR_TO_PLURAL } from '@objectstack/spec/shared'; -import { applyConversionsToStoredItem } from '@objectstack/spec'; +import { applyConversionsToStoredItem, type ConversionNotice } from '@objectstack/spec'; import { type FormView, isAggregatedViewContainer } from '@objectstack/spec/ui'; import { METADATA_FORM_REGISTRY, CORE_SERVICE_PROVIDER, serviceUnavailableMessage } from '@objectstack/spec/system'; -import { DEFAULT_METADATA_TYPE_REGISTRY, getMetadataTypeSchema, getMetadataTypeActions, getMetadataCreateSeed } from '@objectstack/spec/kernel'; +import { DEFAULT_METADATA_TYPE_REGISTRY, getMetadataTypeSchema, getMetadataTypeActions, getMetadataCreateSeed, PROTOCOL_VERSION } from '@objectstack/spec/kernel'; import { extractProtection, evaluateLockForWrite, @@ -49,6 +49,11 @@ import { decorateMetadataItems, type MetadataDiagnostics, } from './metadata-diagnostics.js'; +import type { + StoredMigrationNotice, + StoredMigrationReport, + StoredMigrationRow, +} from './stored-migration.js'; /** * Canonical Zod schema per metadata type lives in @@ -1686,19 +1691,45 @@ export class ObjectStackProtocolImplementation implements */ private convertStoredItem(type: string, data: unknown): unknown { const singular = PLURAL_TO_SINGULAR[type] ?? type; - if (singular === 'flow') return data; - return applyConversionsToStoredItem(singular, data, { + return this.convertStoredItemDetailed(type, data, (n) => { + const name = (data as { name?: unknown } | null | undefined)?.name; + const key = `${n.conversionId}|${singular}|${String(name ?? '')}`; + if (this.storedConversionWarned.has(key)) return; + this.storedConversionWarned.add(key); + console.warn( + `[Protocol] stored ${singular}/${String(name ?? '')} carries a pre-protocol shape; ` + + `${n.message} The row itself is unchanged — re-save it (Studio edit → save, or run ` + + `"os migrate meta --stored --apply") to persist the canonical shape.`, + ); + }).item; + } + + /** + * {@link convertStoredItem} with the chain's notices handed back instead of + * only logged — what {@link migrateStoredMetadata} reports per row (#4327). + * + * The notices ARE the change signal: a conversion emits exactly one per + * rewrite it performs (ADR-0087 D2 "loud"), so an empty list means the row + * is already canonical and there is nothing to persist. Comparing bodies + * instead would be weaker — the pass is copy-on-write, so an untouched + * branch is shared and a re-serialized identical body can still differ in + * key order. + */ + private convertStoredItemDetailed( + type: string, + data: unknown, + onNotice?: (notice: ConversionNotice) => void, + ): { item: unknown; notices: ConversionNotice[] } { + const singular = PLURAL_TO_SINGULAR[type] ?? type; + if (singular === 'flow') return { item: data, notices: [] }; + const notices: ConversionNotice[] = []; + const item = applyConversionsToStoredItem(singular, data, { onNotice: (n) => { - const name = (data as { name?: unknown } | null | undefined)?.name; - const key = `${n.conversionId}|${singular}|${String(name ?? '')}`; - if (this.storedConversionWarned.has(key)) return; - this.storedConversionWarned.add(key); - console.warn( - `[Protocol] stored ${singular}/${String(name ?? '')} carries a pre-protocol shape; ` + - `${n.message} The row itself is unchanged — re-save it (Studio edit → save) to persist the canonical shape.`, - ); + notices.push(n); + onNotice?.(n); }, }); + return { item, notices }; } constructor( @@ -5824,10 +5855,19 @@ export class ObjectStackProtocolImplementation implements return true; } - async saveMetaItem(request: { type: string, name: string, item?: any, organizationId?: string, parentVersion?: string | null, actor?: string, force?: boolean, mode?: 'draft' | 'publish', packageId?: string | null }) { + async saveMetaItem(request: { type: string, name: string, item?: any, organizationId?: string, parentVersion?: string | null, actor?: string, force?: boolean, mode?: 'draft' | 'publish', packageId?: string | null, source?: string }) { if (!request.item) { throw new Error('Item data is required'); } + // What the history row, the audit row and the watch event record as the + // origin of this write. Defaults to this method — the ordinary Studio / + // REST / SDK save. The only caller that overrides it is + // {@link migrateStoredMetadata} (`'migrate-stored'`), so an operator + // reading a diff can tell a canonicalization pass from an author's edit + // (#4327). NOT request-derived: the REST layer builds this request field + // by field and never forwards a client-supplied `source`, so provenance + // stays something the server states, not something a caller claims. + const writeSource = request.source ?? 'protocol.saveMetaItem'; // Drop OUR OWN read decorations before anything reads the body (#4326). // The write path persists verbatim by design (ADR-0005 §Validation), so // the standard Studio round-trip — GET (decorated) → edit → PUT the whole @@ -5890,7 +5930,7 @@ export class ObjectStackProtocolImplementation implements ...(request.organizationId ? { organizationId: request.organizationId } : {}), operation: 'save', ...(request.actor ? { actor: request.actor } : {}), - source: 'protocol.saveMetaItem', + source: writeSource, }); if (lockErr) throw lockErr; } @@ -6148,7 +6188,7 @@ export class ObjectStackProtocolImplementation implements const result = await repo.put(ref, request.item, { parentVersion, actor: request.actor ?? 'system', - source: 'protocol.saveMetaItem', + source: writeSource, intent, state: mode === 'draft' ? 'draft' : 'active', ...(request.packageId !== undefined ? { packageId: request.packageId } : {}), @@ -6171,7 +6211,7 @@ export class ObjectStackProtocolImplementation implements outcome: 'allowed', code: 'ok', ...(request.actor ? { actor: request.actor } : {}), - source: 'protocol.saveMetaItem', + source: writeSource, note: mode === 'draft' ? 'draft' : 'active', }); // [ADR-0094] Awaited projection BEFORE the fire-and-forget @@ -6310,6 +6350,205 @@ export class ObjectStackProtocolImplementation implements } } + /** + * `os migrate meta --stored` — canonicalize `sys_metadata` rows in place so + * the read-path conversion chain has a finish line (#4327). + * + * #4317 made every stored-row rehydration seam replay the full ADR-0087 + * chain, so a row written under any past protocol *reads* canonical forever + * ({@link convertStoredItem}). The rows themselves stayed legacy: the chain + * re-lowers them on every load and each one emits a conversion notice per + * process. This pass ends that for a deployment that runs it — same chain, + * same policy, result written back — while the read path stays the + * guarantee for every deployment that does not (#3855: operator-run + * migrations cannot be relied upon, so nothing here is load-bearing). + * + * ## What it walks + * + * `active` and `draft` rows, every org (the env-wide `organization_id IS + * NULL` bucket included). `archived` / `deprecated` rows are deliberately + * not read: they are not served metadata, and rewriting them would edit a + * record of what *was*. `sys_metadata_history` is untouched for the same + * reason the addendum gives — converting a version body would break the + * checksum↔body pairing. + * + * ## How it writes + * + * Through {@link saveMetaItem}, not the repository directly, so a rewritten + * row gets everything an author's save gets: the schema gate, a + * `sys_metadata_history` row, a fresh checksum, the mutation projectors and + * the watch event Studio's HMR consumes. Three deliberate arguments: + * + * - `parentVersion: row.checksum` — a true optimistic lock. A concurrent + * writer that moved the row between our read and our write gets a 409, + * reported as `failed`, never a clobber. + * - `force: true` — the destructive-change diff compares the *stored* + * body (which `getMetaItem` already serves converted) against the body we + * are about to write (the same conversion). It is empty by construction, + * and there is no author here for a confirmation prompt to reach. + * - `source: 'migrate-stored'` — so a history diff distinguishes a + * canonicalization pass from an edit someone made. + * + * ## What it declines to touch, and says so + * + * - **`flow` rows.** Flow-node conversions carry ADR-0078's open-namespace + * conflict guard, which needs the automation engine's live executor + * registry; this layer does not have it, so flows canonicalize at + * `AutomationEngine.registerFlow` and are reported `skipped` here. + * - **Types with no repository write path** (neither `allowOrgOverride` nor + * `allowRuntimeCreate`). `saveMetaItem` routes those down the legacy + * raw-engine branch, which records no history and forces `state: + * 'active'` — a historyless rewrite that could also promote a draft is + * not what this pass promises, so it declines instead. + * - **Rows that still fail the current schema after conversion.** + * `saveMetaItem` rejects them (422) and that rejection is correct: the + * body is a genuine contract violation, not chain-owned history. They + * surface as `failed` with the validation message, keep reading through + * the chain, and stay fixable in Studio. + */ + async migrateStoredMetadata(request: { + /** Write. Omitted / false = preview: reports what it would do, writes nothing. */ + apply?: boolean; + /** Restrict to these metadata types (singular or plural spelling). */ + types?: string[]; + /** Recorded as the writer on the history + audit rows. */ + actor?: string; + } = {}): Promise { + const apply = request.apply === true; + const typeFilter = request.types && request.types.length > 0 + ? new Set(request.types.map((t) => PLURAL_TO_SINGULAR[t] ?? t)) + : null; + + const report: StoredMigrationReport = { + apply, + protocol: PROTOCOL_VERSION, + scanned: 0, + canonical: 0, + pending: 0, + rewritten: 0, + skipped: 0, + failed: 0, + rows: [], + }; + + // Two scoped queries rather than one unfiltered scan: `state` is an + // equality column and these are the only two states that are SERVED + // metadata. Archived bodies are never even read. + const rows: any[] = []; + for (const state of ['active', 'draft'] as const) { + rows.push(...await this.engine.find('sys_metadata', { where: { state } })); + } + + for (const row of rows) { + const rawType = String(row.type ?? ''); + const singular = PLURAL_TO_SINGULAR[rawType] ?? rawType; + if (typeFilter && !typeFilter.has(singular)) continue; + report.scanned++; + + const state: 'active' | 'draft' = row.state === 'draft' ? 'draft' : 'active'; + const organizationId: string | null = row.organization_id ?? null; + const packageId: string | null = row.package_id ?? null; + const base = { + id: String(row.id ?? ''), + type: singular, + name: String(row.name ?? ''), + organizationId, + packageId, + state, + notices: [] as StoredMigrationNotice[], + }; + // An already-canonical row is counted, never itemised: on a healthy + // deployment that is every row, and a report listing all of them + // would bury the handful that actually need something. + const record = (entry: StoredMigrationRow): void => { + if (entry.outcome === 'canonical') { + report.canonical++; + return; + } + report[entry.outcome]++; + report.rows.push(entry); + }; + + let body: unknown; + try { + body = typeof row.metadata === 'string' ? JSON.parse(row.metadata) : row.metadata; + } catch (e: any) { + record({ + ...base, + outcome: 'failed', + reason: `the stored body is not valid JSON (${e?.message ?? String(e)})`, + }); + continue; + } + + if (singular === 'flow') { + record({ + ...base, + outcome: 'skipped', + reason: 'flows canonicalize at AutomationEngine.registerFlow — the node-type ' + + 'conflict guard needs the live executor registry this layer does not have', + }); + continue; + } + + const overlayAllowed = ObjectStackProtocolImplementation.isOverlayAllowed(singular); + const runtimeCreateAllowed = ObjectStackProtocolImplementation.isRuntimeCreateAllowed(singular); + if (!overlayAllowed && !runtimeCreateAllowed) { + record({ + ...base, + outcome: 'skipped', + reason: `type '${singular}' has no repository write path (allowOrgOverride and ` + + 'allowRuntimeCreate are both false), so a rewrite would record no history', + }); + continue; + } + + const { item, notices } = this.convertStoredItemDetailed(singular, body); + if (notices.length === 0) { + record({ ...base, outcome: 'canonical' }); + continue; + } + const flattened: StoredMigrationNotice[] = notices.map((n) => ({ + conversionId: n.conversionId, + surface: n.surface, + from: n.from, + to: n.to, + path: n.path, + message: n.message, + })); + + if (!apply) { + record({ ...base, notices: flattened, outcome: 'pending' }); + continue; + } + + try { + await this.saveMetaItem({ + type: singular, + name: base.name, + item, + mode: state === 'draft' ? 'draft' : 'publish', + parentVersion: row.checksum ?? null, + packageId, + force: true, + source: 'migrate-stored', + actor: request.actor ?? 'migrate-stored', + ...(organizationId ? { organizationId } : {}), + }); + record({ ...base, notices: flattened, outcome: 'rewritten' }); + } catch (e: any) { + record({ + ...base, + notices: flattened, + outcome: 'failed', + reason: e?.message ?? String(e), + }); + } + } + + return report; + } + /** * Yield the durable change-log for a single metadata item — every * put/delete recorded in `sys_metadata_history` for `(org, type, name)`, diff --git a/packages/metadata-protocol/src/stored-migration.ts b/packages/metadata-protocol/src/stored-migration.ts new file mode 100644 index 0000000000..0cccec8a19 --- /dev/null +++ b/packages/metadata-protocol/src/stored-migration.ts @@ -0,0 +1,180 @@ +// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license. + +/** + * The report shape + renderer for the stored-metadata canonicalization pass + * (#4327), the follow-up to #3903's read-path guarantee. + * + * #4317 closed the correctness gap from the read side: every stored-row + * rehydration seam replays the full ADR-0087 conversion chain + * (`applyConversionsToStoredItem`, retired entries included), so a row written + * under any past protocol is *served* canonical forever. What it deliberately + * did not do is make the rows themselves canonical — a pre-17 row keeps its + * legacy bytes, the chain re-lowers it on every load, and each affected row + * emits one conversion notice per process. + * + * {@link ObjectStackProtocolImplementation.migrateStoredMetadata} is the + * operator-run pass that ends that: same chain, same policy, but the result is + * written back through the normal write path so the row stops carrying the old + * dialect. This module owns the vocabulary that pass reports in, kept beside + * the types rather than in the CLI so an admin route renders the same run the + * same way. + * + * **Not load-bearing.** #3855's conclusion still holds — an operator-run + * migration cannot be relied on, so the read path stays the guarantee and + * nothing gates on this having run. No `sys_migration` flag is recorded for + * exactly that reason: a flag row would advertise a gate that does not exist. + * The verifiable statement operators wanted is the *re-run* — a second pass + * that reports every row canonical is the evidence, and it costs one command. + */ + +/** What the pass did with (or would do with) one `sys_metadata` row. */ +export type StoredMigrationOutcome = + /** The chain was a no-op — the row is already on protocol. Not itemised. */ + | 'canonical' + /** Preview: the chain would rewrite this row. Nothing was written. */ + | 'pending' + /** Apply: the canonical body was re-saved through the write path. */ + | 'rewritten' + /** Outside this pass's reach — see {@link StoredMigrationRow.reason}. */ + | 'skipped' + /** The row could not be read, or the re-save was refused. */ + | 'failed'; + +/** + * One conversion the chain applied to a row — the per-row detail a preview + * run prints, flattened from the spec's {@link ConversionNotice} to the + * fields an operator acts on. + */ +export interface StoredMigrationNotice { + /** The `MetadataConversion.id` that fired, e.g. `flow-node-crud-filter-alias`. */ + conversionId: string; + /** Dotted surface the conversion governs, e.g. `object.field.conditionalRequired`. */ + surface: string; + /** The off-spec token/shape found in the stored body. */ + from: string; + /** The canonical token/shape it was lowered to. */ + to: string; + /** Where in the item it applied, e.g. `objects[0].fields.amount`. */ + path: string; + /** The chain's own human-facing line. */ + message: string; +} + +/** Per-row result. Rows the chain left alone (`canonical`) are counted, not listed. */ +export interface StoredMigrationRow { + /** `sys_metadata.id` — the row this is about, so an operator can go look at it. */ + id: string; + /** Singular metadata type (`object`, `view`, …), normalized from the row's spelling. */ + type: string; + name: string; + /** `null` = the env-wide overlay bucket. */ + organizationId: string | null; + /** `null` = a package-less (global) overlay row. */ + packageId: string | null; + state: 'active' | 'draft'; + outcome: StoredMigrationOutcome; + /** The conversions this row carries. Empty unless the chain rewrote something. */ + notices: StoredMigrationNotice[]; + /** Why a `skipped` / `failed` row was not rewritten. Absent otherwise. */ + reason?: string; +} + +/** The whole run. `apply: false` is a preview — it writes nothing, by construction. */ +export interface StoredMigrationReport { + /** False = preview. A preview never writes, not even a row it would leave identical. */ + apply: boolean; + /** The protocol version the chain canonicalized *to* — what a clean run attests. */ + protocol: string; + /** Rows examined (after any `--type` filter; archived/deprecated rows are not read). */ + scanned: number; + /** Rows the chain left unchanged — already on protocol. */ + canonical: number; + /** Preview only: rows the chain would rewrite. Always 0 on an apply run. */ + pending: number; + /** Apply only: rows re-saved through the write path. */ + rewritten: number; + skipped: number; + failed: number; + /** Every row that is not `canonical`, in scan order. */ + rows: StoredMigrationRow[]; +} + +/** + * Is this deployment's stored metadata on protocol? + * + * True when nothing is left to convert and nothing was refused. `skipped` rows + * deliberately do NOT count against it: they name a seam that owns them + * (flows canonicalize at `AutomationEngine.registerFlow`) or a type this pass + * has no history-recording write path for — both are documented carve-outs, not + * work this command left half-done. They are still printed, so the operator + * sees what the verdict does not cover. + */ +export function storedMigrationClean(report: StoredMigrationReport): boolean { + return report.pending === 0 && report.failed === 0; +} + +/** Render a run for a terminal. One line per non-canonical row, notices nested. */ +export function formatStoredMigrationReport(report: StoredMigrationReport): string[] { + const lines: string[] = []; + lines.push( + `Examined ${report.scanned} stored metadata row(s) (active + draft, all orgs) ` + + `against the protocol ${report.protocol} conversion chain.`, + ); + + const converting = report.rows.filter((r) => r.outcome === 'pending' || r.outcome === 'rewritten'); + if (converting.length > 0) { + lines.push( + report.apply + ? `✓ Rewrote ${report.rewritten} row(s) carrying a pre-protocol shape:` + : `→ ${report.pending} row(s) carry a pre-protocol shape and would be rewritten:`, + ); + for (const row of converting) { + lines.push(` • ${row.type}/${row.name} ${describeScope(row)}`); + for (const n of row.notices) { + lines.push(` ${n.conversionId}: ${n.from} → ${n.to} at ${n.path}`); + } + } + } + + const skipped = report.rows.filter((r) => r.outcome === 'skipped'); + if (skipped.length > 0) { + lines.push(`⚠ ${skipped.length} row(s) are outside this pass — they keep reading through the chain:`); + for (const row of skipped) { + lines.push(` • ${row.type}/${row.name} ${describeScope(row)} — ${row.reason ?? 'skipped'}`); + } + } + + const failed = report.rows.filter((r) => r.outcome === 'failed'); + if (failed.length > 0) { + lines.push(`✗ ${failed.length} row(s) could not be rewritten:`); + for (const row of failed) { + lines.push(` • ${row.type}/${row.name} ${describeScope(row)} — ${row.reason ?? 'failed'}`); + } + } + + if (report.scanned === 0) { + // "Nothing to convert" and "nothing was looked at" are different claims, + // and only the first is a pass. A run pointed at the wrong project — the + // database line above names which one — would otherwise read as clean. + lines.push( + '⚠ No rows were examined, so this run attests nothing. An empty result is what ' + + 'a deployment that has never authored metadata looks like, and also what running ' + + 'from the wrong project root looks like — check the database named above.', + ); + } else if (converting.length === 0 && failed.length === 0) { + lines.push( + `✓ Every row examined is already on protocol ${report.protocol} — ` + + 'the read-path conversion pass is a no-op here.', + ); + } + return lines; +} + +/** `[org=… package=… draft]` — only the parts that are not the default. */ +function describeScope(row: StoredMigrationRow): string { + const parts: string[] = []; + parts.push(row.organizationId ? `org=${row.organizationId}` : 'env-wide'); + if (row.packageId) parts.push(`package=${row.packageId}`); + if (row.state === 'draft') parts.push('draft'); + return `[${parts.join(', ')}]`; +} diff --git a/pnpm-lock.yaml b/pnpm-lock.yaml index 6c7903a29f..b7f6b44014 100644 --- a/pnpm-lock.yaml +++ b/pnpm-lock.yaml @@ -402,6 +402,9 @@ importers: '@objectstack/metadata': specifier: workspace:* version: link:../metadata + '@objectstack/metadata-protocol': + specifier: workspace:* + version: link:../metadata-protocol '@objectstack/objectql': specifier: workspace:^ version: link:../objectql From 98394febb4f988511ae92bff7e49e6f4884e546e Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 08:09:32 +0000 Subject: [PATCH 2/3] docs(adr-0087): name the follow-up that gives flow rows the same finish line (#4454) The addendum said flows were "tracked separately" without saying where. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WoZPKPDqJ7WB7z84xk9y3f --- docs/adr/0087-metadata-protocol-upgrade-contract.md | 5 +++-- 1 file changed, 3 insertions(+), 2 deletions(-) diff --git a/docs/adr/0087-metadata-protocol-upgrade-contract.md b/docs/adr/0087-metadata-protocol-upgrade-contract.md index 973574f8cd..9c98420c8f 100644 --- a/docs/adr/0087-metadata-protocol-upgrade-contract.md +++ b/docs/adr/0087-metadata-protocol-upgrade-contract.md @@ -454,5 +454,6 @@ writing mode. `AutomationEngine.registerFlow`, which holds the executor registry the conflict guard needs) and types with no repository write path are reported as `skipped` with the reason, never counted as done. Giving flows the same finish - line needs a canonicalization entry point on the automation engine and is - tracked separately. + line needs a canonicalization entry point on the automation engine — tracked + as #4454, and worth doing precisely because the graduated flow-node + conversions are where the most stored dialect lives. From 66a2db94841b9a99e683041b48fcaaec1ef50737 Mon Sep 17 00:00:00 2001 From: Claude Date: Sat, 1 Aug 2026 08:31:45 +0000 Subject: [PATCH 3/3] docs(releases): name the stored-metadata pass on the v17 page (#4327) MIME-Version: 1.0 Content-Type: text/plain; charset=UTF-8 Content-Transfer-Encoding: 8bit The v17 entry for #3903 described the read-path guarantee and stopped there, and the upgrade checklist listed the two per-deployment migrations without this one. Both now point at `os migrate meta --stored`, marked optional — it opens no gate, unlike its two neighbours in that list. Co-Authored-By: Claude Opus 5 Claude-Session: https://claude.ai/code/session_01WoZPKPDqJ7WB7z84xk9y3f --- content/docs/releases/v17.mdx | 13 ++++++++++++- 1 file changed, 12 insertions(+), 1 deletion(-) diff --git a/content/docs/releases/v17.mdx b/content/docs/releases/v17.mdx index 3c3ac3da59..a5b9e5ebee 100644 --- a/content/docs/releases/v17.mdx +++ b/content/docs/releases/v17.mdx @@ -1672,7 +1672,12 @@ platform capabilities an administrator gains, then the Console delta. source at every read seam (including `registerFlow` rehydration) — a stored action with the removed `execute` dispatches via `target` again; boot hydration validates each row post-conversion and diagnoses invalid ones - with `[metadata_spec_invalid]` instead of shrugging. + with `[metadata_spec_invalid]` instead of shrugging. **The rows themselves + can now be brought forward too (#4327):** `os migrate meta --stored` + replays the chain over `sys_metadata` and rewrites what still carries a + pre-protocol shape, through the normal write path. Optional — the read path + is the guarantee either way — but it is what makes the conversion pass a + no-op on your data instead of a permanent shim. - **Studio's metadata forms tell the truth (#3786).** Four of seventeen forms had drifted from their schemas, so controls saved nothing: the Object → Capabilities toggles bound a key the schema does not declare (all seven @@ -1977,6 +1982,12 @@ covers are folded into the list below rather than left to the changelog.) fix what it reports before `--apply`. It converts nothing — the values it names are application data — and a scan that was truncated or could not read an object fails the gate even at zero violations. +- **Stored metadata (optional):** run `os migrate meta --stored` to see which + `sys_metadata` rows still carry a pre-17 shape, and `--apply` to rewrite + them. Unlike the two above this opens no gate and nothing depends on it — + those rows already read canonically, forever. It stops them re-converting on + every load, keeps diffs and exports clean going forward, and gives you an + exit code to assert on: nothing left to do exits `0`. - **Datasources:** verify every declared datasource connects in every environment — a bound datasource that cannot connect now fails the boot instead of failing every later query.