feat(migrate,metadata-protocol): os migrate meta --stored rewrites sys_metadata rows in place (#4327)

.changeset/migrate-meta-stored-rewrite.md

@@ -0,0 +1,62 @@
+---
+"@objectstack/metadata-protocol": minor
+"@objectstack/cli": patch
+---
+
+feat(migrate,metadata-protocol): `os migrate meta --stored` rewrites sys_metadata rows so the read-path chain has a finish line (#4327)
+
+#4317 closed the correctness gap from the read side: every stored-row
+rehydration seam replays the full ADR-0087 conversion chain, retired entries
+included, so a row written under any past protocol is *served* canonical
+forever. What it deliberately did not do is make the rows themselves canonical.
+A pre-17 row keeps its legacy bytes, the chain re-lowers it on every load, and
+each affected row logs one conversion notice per process — deduped, but back
+every boot. Until now the only things that ever rewrote such a row were a Studio
+re-save and `duplicatePackage`.
+
+**`os migrate meta --stored`** is the pass that ends it for a deployment that
+runs it. It walks `sys_metadata` — `active` and `draft`, every organization —
+replays the same `applyConversionsToStoredItem` chain, and re-saves each changed
+body through the normal write path, so a rewritten row gets a
+`sys_metadata_history` entry, a fresh checksum and the mutation projectors,
+exactly like an author's save. The history row's `source` is `migrate-stored`,
+so a later diff distinguishes an upgrade from somebody's edit.
+
+```bash
+os migrate meta --stored                    # preview: per-row report, writes nothing
+os migrate meta --stored --apply            # rewrite the rows (prompts)
+os migrate meta --stored --apply --yes --json   # CI / scripts
+os migrate meta --stored --type view        # restrict to a type (repeatable)
+```
+
+**Preview is the default and `--apply` is the only writing mode** — the house
+rule its siblings already keep (#3617's "a dry run changes nothing"), and it
+applies with more force here because what moves is metadata: every affected
+row's checksum and a history entry per row. An apply run also refuses to start
+while another process holds the SQLite database, for the same reason
+`os migrate files-to-references --apply` does.
+
+**Nothing gates on this having run.** #3855's conclusion stands — an
+operator-run migration cannot be relied upon, so the read path remains the
+guarantee for every deployment, and no `sys_migration` flag is recorded (a flag
+would advertise enforcement that does not exist). What a run buys is hygiene —
+rows stop carrying pre-protocol dialects, so diffs, exports and history are
+clean going forward, and the recurring notices go quiet — plus one thing that
+was previously unobtainable: **an operator can assert it.** A run with nothing
+left to do exits `0`, a deployment with rows still on an old dialect exits `1`,
+so "my metadata is on protocol N" becomes a CI check rather than a belief.
+
+Three things the pass declines, and reports rather than counting as done:
+`flow` rows (their seam is `AutomationEngine.registerFlow`, which holds the
+executor registry the node-type conflict guard needs), types with no repository
+write path (`agent` — rewriting there would record no history and force a draft
+live), and rows that still fail the current schema after conversion (a genuine
+contract violation the write path is right to refuse; it keeps reading through
+the chain and stays fixable in Studio).
+
+Also new, and usable without the CLI: `protocol.migrateStoredMetadata()` returns
+the same structured report an admin route would render, and `saveMetaItem`
+accepts an optional `source` for the history/audit rows. `source` is not
+request-derived — the REST layer builds its save request field by field and
+never forwards a client-supplied value, so provenance stays something the server
+states rather than something a caller claims.

content/docs/deployment/cli.mdx

@@ -589,6 +589,7 @@ written.
 | `os migrate plan` | Warns and continues — a plan writes nothing either way |
 | `os migrate apply` | **Refuses** (exit 1, `error: database_busy` under `--json`). Stop the other process, or pass `--force` |
 | `os migrate files-to-references --apply` | **Refuses** likewise — it rewrites rows, so a concurrent writer is at least as dangerous |
+| `os migrate meta --stored --apply` | **Refuses** likewise — it rewrites `sys_metadata` rows, and a live process saving metadata is exactly the collision |
 
 The check applies to SQLite only: Postgres and MySQL take their own server-side
 locks. Only same-user processes are visible without elevated privileges, and a
@@ -642,6 +643,7 @@ where the data lives.
 |---------|-------------|
 | `os migrate files-to-references` | Convert legacy file-field values to `sys_file` references, verify the ownership ledger, and record the deployment's migration flag |
 | `os migrate value-shapes` | Scan stored reference and structured-JSON field values against the platform's value contract, and record the deployment's migration flag when clean |
+| `os migrate meta --stored` | Replay the metadata conversion chain over this deployment's `sys_metadata` rows and rewrite the ones still carrying a pre-protocol shape. Hygiene, not a gate — nothing depends on it having run |
 
 ```bash
 os migrate files-to-references            # Dry run: full report, writes nothing
@@ -774,6 +776,64 @@ closed gate logs that it is enforcing, and an app that declares neither class
 of field says nothing at all. So a running deployment always tells you the
 state of its own data — which is the question `os migrate meta` cannot answer.
 
+#### `os migrate meta --stored`
+
+The two commands above are about **application data**. This one is about the
+**metadata itself**, at rest: the `sys_metadata` rows Studio and the runtime
+authoring APIs write.
+
+Those rows already *read* correctly whatever protocol they were written under —
+every rehydration seam replays the full conversion chain, so a body from an
+older major is served in today's canonical shape and always will be. What the
+rows do not do is *change*: they keep their original bytes, the chain re-lowers
+them on every load, and each one logs a conversion notice once per boot. This
+command ends that for the deployment that runs it.
+
+```bash
+os migrate meta --stored                    # Preview: per-row report, writes nothing
+os migrate meta --stored --apply            # Rewrite the rows (prompts)
+os migrate meta --stored --apply --yes --json   # CI / scripts
+os migrate meta --stored --type view --type object   # Restrict to a type (repeatable)
+```
+
+It walks `active` and `draft` rows across every organization (archived rows are
+a record of what *was* and are never read), replays the same chain the read path
+does, and re-saves each changed body through the normal write path — so a
+rewritten row gets a `sys_metadata_history` entry, a fresh checksum, and the
+mutation projectors, exactly like an author's save. The history entry's source
+is `migrate-stored`, so a later diff shows which changes were an upgrade and
+which were somebody's edit.
+
+Three things it deliberately declines, and names in the report rather than
+counting as done:
+
+| Not rewritten | Why |
+| :--- | :--- |
+| `flow` rows | Flow-node conversions carry a conflict guard that needs the automation engine's live executor registry; flows canonicalize at their own seam when the engine loads them |
+| Types with no repository write path (`agent`) | Their write path records no history and would force a draft live — a half-write is worse than leaving the row to the read path |
+| Rows that still fail the current schema after conversion | That is a genuine contract violation, not chain-owned history. The write path's rejection is correct; fix the row in Studio |
+
+<Callout type="warn">
+`--apply` is the only writing mode, and it rewrites **metadata** — each affected
+row's checksum moves and each gets a history entry. Preview first. Like the
+other row-rewriting migration, an apply run refuses to start while another
+process holds the SQLite database (`--force` overrides).
+</Callout>
+
+**Nothing gates on this having run.** The read path is the guarantee, for every
+deployment, whether or not anyone runs this — an operator-run migration is not
+something the platform can depend on. What running it buys is hygiene (cleaner
+diffs, exports and history from here on, and the recurring boot notices go
+quiet) plus one thing that was previously unobtainable: **you can assert it.**
+A run with nothing left to do exits `0`; a deployment with rows still carrying
+an old dialect exits `1`. So "my metadata is on protocol N" becomes a check
+rather than a belief.
+
+Note the division of labour with the default mode: `os migrate meta --from N`
+rewrites an **author's source** and reads no database; `--stored` rewrites **one
+deployment's rows** and reads no config. Same chain, opposite ends of the
+contract — which is why the two modes are mutually exclusive.
+
 ### Scaffolding
 
 | Command | Alias | Description |

content/docs/releases/v17.mdx

@@ -1672,7 +1672,12 @@ platform capabilities an administrator gains, then the Console delta.
   source at every read seam (including `registerFlow` rehydration) — a stored
   action with the removed `execute` dispatches via `target` again; boot
   hydration validates each row post-conversion and diagnoses invalid ones
-  with `[metadata_spec_invalid]` instead of shrugging.
+  with `[metadata_spec_invalid]` instead of shrugging. **The rows themselves
+  can now be brought forward too (#4327):** `os migrate meta --stored`
+  replays the chain over `sys_metadata` and rewrites what still carries a
+  pre-protocol shape, through the normal write path. Optional — the read path
+  is the guarantee either way — but it is what makes the conversion pass a
+  no-op on your data instead of a permanent shim.
 - **Studio's metadata forms tell the truth (#3786).** Four of seventeen forms
   had drifted from their schemas, so controls saved nothing: the Object →
   Capabilities toggles bound a key the schema does not declare (all seven
@@ -1977,6 +1982,12 @@ covers are folded into the list below rather than left to the changelog.)
   fix what it reports before `--apply`. It converts nothing — the values it
   names are application data — and a scan that was truncated or could not read
   an object fails the gate even at zero violations.
+- **Stored metadata (optional):** run `os migrate meta --stored` to see which
+  `sys_metadata` rows still carry a pre-17 shape, and `--apply` to rewrite
+  them. Unlike the two above this opens no gate and nothing depends on it —
+  those rows already read canonically, forever. It stops them re-converting on
+  every load, keeps diffs and exports clean going forward, and gives you an
+  exit code to assert on: nothing left to do exits `0`.
 - **Datasources:** verify every declared datasource connects in every
   environment — a bound datasource that cannot connect now fails the boot
   instead of failing every later query.

docs/adr/0087-metadata-protocol-upgrade-contract.md

@@ -421,3 +421,39 @@ diverged (#3903). This addendum extends the contract to data at rest:
   at boot would unhook live tables and make the row unfixable in Studio
   (availability over purity for data at rest; the same verdict reaches Studio
   as `_diagnostics` on every read).
+
+## Addendum (2026-08-01) — the stored chain gets a finish line (#4327)
+
+The addendum above makes a legacy row read canonical *forever*, which is the
+correctness guarantee — and, read literally, also a promise that the chain runs
+on that row forever. `os migrate meta --stored`
+(`ObjectStackProtocolImplementation.migrateStoredMetadata`) lets a deployment
+end that for itself: it walks `sys_metadata` (active + draft, all orgs), replays
+the same `applyConversionsToStoredItem` pass, and re-saves each changed body
+through `saveMetaItem` with `source: 'migrate-stored'` — history row, checksum,
+mutation projectors and all. Preview is the default; `--apply` is the only
+writing mode.
+
+- **Not load-bearing, and no flag.** #3855's conclusion stands: an operator-run
+  migration cannot be relied on, so the read path — not this — remains the
+  guarantee, and nothing gates on it having run. Deliberately no `sys_migration`
+  row either: unlike ADR-0104's two gates, a flag here would advertise
+  enforcement that does not exist. The verifiable statement operators wanted is
+  the **re-run** — a second pass reporting every row canonical exits 0, so "my
+  metadata is on protocol N" is a check rather than a belief.
+- **The write path's gate is not bypassed.** A body that still fails the current
+  schema after conversion is refused (422) and reported, exactly as the bullet
+  above describes for reads: it is a genuine contract violation, and the pass
+  has no more standing to persist it than an author does. It keeps reading
+  through the chain and stays fixable in Studio.
+- **The version layer stays verbatim.** `sys_metadata_history` is appended to,
+  never rewritten. Canonicalizing a past version's body would break the
+  checksum↔body pairing this contract depends on — the migration is a new
+  commit, not a rewrite of history.
+- **What the pass does not cover, it names.** Flows (their seam is
+  `AutomationEngine.registerFlow`, which holds the executor registry the
+  conflict guard needs) and types with no repository write path are reported as
+  `skipped` with the reason, never counted as done. Giving flows the same finish
+  line needs a canonicalization entry point on the automation engine — tracked
+  as #4454, and worth doing precisely because the graduated flow-node
+  conversions are where the most stored dialect lives.

packages/cli/package.json

@@ -55,6 +55,7 @@
     "@objectstack/lint": "workspace:*",
     "@objectstack/mcp": "workspace:*",
     "@objectstack/metadata": "workspace:*",
+    "@objectstack/metadata-protocol": "workspace:*",
     "@objectstack/objectql": "workspace:^",
     "@objectstack/observability": "workspace:^",
     "@objectstack/platform-objects": "workspace:*",

packages/cli/src/commands/migrate/meta.stored-flags.test.ts

@@ -0,0 +1,68 @@
+// Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
+
+/**
+ * #4327 — the flag surface where `os migrate meta`'s two modes meet.
+ *
+ * One command, two subjects: `--from N` replays the ADR-0087 chain over an
+ * **author's source** (a config file, no database in reach), `--stored` replays
+ * it over **one deployment's** `sys_metadata` rows (a database, no config in
+ * reach). Everything below pins a boundary that a plausible-looking oclif
+ * declaration gets wrong.
+ */
+import { describe, expect, it } from 'vitest';
+import MigrateMeta, { storedOnlyFlagsIn } from './meta.js';
+
+describe('storedOnlyFlagsIn (#4327)', () => {
+  it('reports the stored-only flags the operator typed', () => {
+    expect(storedOnlyFlagsIn(['--from', '16', '--apply'])).toEqual(['apply']);
+    expect(storedOnlyFlagsIn(['--from=16', '--type=view', '--force'])).toEqual(['force', 'type']);
+    expect(storedOnlyFlagsIn(['--from', '16', '-y'])).toEqual(['yes']);
+  });
+
+  it('says nothing about a run that typed none of them', () => {
+    expect(storedOnlyFlagsIn(['--from', '16', '--step', '--json'])).toEqual([]);
+    expect(storedOnlyFlagsIn([])).toEqual([]);
+  });
+
+  it('never double-reports --yes given both spellings', () => {
+    expect(storedOnlyFlagsIn(['--stored', '--yes', '-y'])).toEqual(['yes']);
+  });
+
+  it('reads argv, not the environment — an exported OS_DATABASE_URL is not a typed flag', () => {
+    // The trap that made `dependsOn` unusable: oclif fills `--database-url`
+    // from `OS_DATABASE_URL`, so a merely-exported env var would have counted
+    // as "provided" and broken `os migrate meta --from N` for anyone who has
+    // one set. Provenance comes from argv precisely so it cannot.
+    expect(storedOnlyFlagsIn(['--from', '16'])).toEqual([]);
+    expect(storedOnlyFlagsIn(['--stored', '--database-url', 'sqlite://x.db'])).toEqual(['database-url']);
+  });
+});
+
+describe('MigrateMeta flag declarations (#4327)', () => {
+  const flags = MigrateMeta.flags as Record<string, any>;
+
+  it('does not declare --from required — that would reject every --stored run', () => {
+    expect(flags.from.required).not.toBe(true);
+  });
+
+  it('makes the authored-chain flags exclusive with --stored', () => {
+    // `--from` names a major an author wrote against; a stored row carries its
+    // own history and gets the full chain regardless. Accepting both would
+    // imply the stored pass honours a range it does not have.
+    for (const name of ['from', 'to', 'step', 'out']) {
+      expect(flags[name].exclusive).toContain('stored');
+    }
+  });
+
+  it('leaves the stored-only flags free of dependsOn', () => {
+    // Enforced by `storedOnlyFlagsIn` instead: see the env-var case above.
+    for (const name of ['apply', 'yes', 'force', 'type', 'database-url']) {
+      expect(flags[name].dependsOn).toBeUndefined();
+    }
+  });
+
+  it('defaults --apply off — preview is the only mode a bare run can have', () => {
+    expect(flags.apply.default).toBe(false);
+    expect(flags.stored.default).toBe(false);
+  });
+});