Merge remote-tracking branch 'origin/main' into claude/rest-list-field-validation-yyx9tk

# Conflicts:
#	packages/objectql/src/engine.ts

Author: claude

.changeset/adr-0104-attestation-adr-note.md

@@ -0,0 +1,4 @@
+---
+---
+
+docs(adr-0104): record the fresh-datastore attestation seam as implemented — driver-reported created-vs-found table counts, not "sys_migration is missing" (which is the 16→17 trap) — and note that born-attested dogfood now exercises the #3459 collection path on every boot. Releases nothing.

.changeset/adr-0104-fresh-datastore-attestation.md

@@ -0,0 +1,47 @@
+---
+"@objectstack/spec": minor
+"@objectstack/platform-objects": minor
+"@objectstack/objectql": minor
+"@objectstack/driver-memory": minor
+"@objectstack/driver-sql": minor
+"@objectstack/service-storage": minor
+---
+
+feat(migrate): a datastore created from empty attests its data migrations at creation (#3438, ADR-0104 2026-07-30 addendum)
+
+Deployment-level migration flags could only be recorded by running
+`os migrate`. That left a hole at the other end of a deployment's life: a
+database created on a version that already ships the migrations started **lax**
+and stayed lax until someone thought to run a command that, for them, converts
+nothing and finds nothing. Every new deployment re-entered the warn regime, so
+the warn regime would never die out — and, since #3459, every new deployment
+also kept every released file forever.
+
+A store the platform **creates from empty** now records
+`adr-0104-file-references` and `adr-0104-value-shapes` at that moment. Nothing
+to run; enforcement and collection are live from the first boot.
+
+**This is not version-gating in disguise.** The fact recorded — no legacy value
+is stored here — is *observed*: the store had no history at all. The platform
+attests only what it watched itself create, and the test is deliberately
+strict: every table made by this boot and **none found already present**. One
+pre-existing table anywhere, one datasource that was already there, one driver
+that cannot account for its schema sync — any of those and the deployment
+attests nothing and produces its evidence by scan, exactly as before. "Found
+empty" and "created empty" are not the same claim, and only the second is an
+observation.
+
+**New surfaces.** `IDataDriver.getSchemaSyncStats?()` (optional, purely
+observational: tables created vs found since connect — implemented by the SQL
+and in-memory drivers), `engine.wasDatastoreCreatedFromEmpty()`,
+`attestFreshDatastore()` in `@objectstack/platform-objects/system`, and
+`VALUE_SHAPES_MIGRATION_ID` / `CREATION_ATTESTED_MIGRATION_IDS` in
+`@objectstack/spec/system`. Attestation never overwrites an existing flag row
+and never throws into a boot: a failure leaves the deployment lax, which a
+migration run can still fix.
+
+**Upgrading changes nothing for an existing database.** It is non-empty when
+the platform reaches it, so it is never attested — run
+`os migrate files-to-references --apply` as before. Importing legacy values
+into an attested deployment is rejected loudly at the write path;
+`OS_ALLOW_LAX_MEDIA_VALUES=1` re-opens leniency while you diagnose.

.changeset/adr-0104-value-shapes-scan-gate.md

@@ -0,0 +1,62 @@
+---
+"@objectstack/spec": minor
+"@objectstack/objectql": minor
+"@objectstack/cli": minor
+---
+
+feat(migrate): `os migrate value-shapes` — the per-deployment gate for reference and structured-JSON value shapes (#3438)
+
+The second of ADR-0104 D1's two evidence gates. Media value shapes already
+enforce once a deployment has verified its file migration (#3681); the
+reference (`lookup` / `master_detail` / `user` / `tree`) and structured-JSON
+(`location` / `address` / `composite` / `repeater` / `record` / `vector`)
+classes now get a gate of their own.
+
+```bash
+os migrate value-shapes           # scan: reports, writes nothing
+os migrate value-shapes --apply   # scan + record the deployment flag when clean
+```
+
+The run walks every stored value of those classes against
+`valueSchemaFor(field, 'stored')` — the same predicate the write path enforces,
+imported rather than re-derived — and, at zero violations, records
+`sys_migration { id: 'adr-0104-value-shapes', verified_at, blocking: 0 }`.
+Strict enforcement of these classes reads **that row**, never the platform
+version, so upgrading changes nothing until a deployment produces its own
+evidence.
+
+**There is no backfill, deliberately.** The file migration converts legacy
+values because the platform narrowed that storage form and owes the conversion.
+A malformed `location` is application data whose correct value only its author
+knows, so this run reports and prescribes — naming the object, field, type,
+count, offending record ids and the parse issue — and the operator fixes and
+re-runs. With nothing to convert, `--apply`'s only write is the flag row, which
+keeps the #3617 invariant trivially: a dry run changes nothing, and whether a
+run changed this deployment's posture never depends on what it found.
+
+**A separate flag from the file migration**, because it attests a separate
+fact. That flag says file values were migrated and their ownership reconciled;
+it says nothing about whether a `lookup` id or a `location` payload is well
+formed. Gating these classes on it would be borrowing evidence for a fact it
+does not cover.
+
+- New escape hatch **`OS_ALLOW_LAX_VALUE_SHAPES=1`** returns a verified
+  deployment to warnings, with the same precedence as its media sibling: the
+  opt-out beats `OS_DATA_VALUE_SHAPE_STRICT_ENABLED`, which beats the flag.
+  Wrongly staying lenient costs a warning; wrongly enforcing stops a working
+  app from writing.
+- `@objectstack/spec/system` exports `VALUE_SHAPES_MIGRATION_ID`.
+- `@objectstack/objectql` exports `scanValueShapes`, `valueShapeScanPassed`
+  and `formatValueShapeScanReport`. The scanner is read-only and does **not**
+  record the flag: readers of a migration flag use the spec contract, only
+  writers depend on `@objectstack/platform-objects`, so the composition lives
+  with the CLI command rather than inverting the engine's dependencies.
+- `validateRecord` gains `valueShapeStrict`, the sibling of
+  `mediaValueShapeStrict`. Both default to `false`: a caller that cannot say
+  stays lenient, so nothing starts rejecting merely because the evidence was
+  unavailable.
+
+**Nothing changes for an existing deployment until it runs the command.** A
+scan that is truncated, or that cannot read an object, fails the gate even with
+zero violations found — "none in the part we read" is not the claim the flag
+makes.

.changeset/builtin-node-config-reconciliation.md

@@ -0,0 +1,47 @@
+---
+'@objectstack/spec': minor
+'@objectstack/service-automation': minor
+---
+
+Reconcile the remaining flat builtins' declared config against what their
+executors read (#4045 — the CRUD / screen / map step, after notify / http /
+connector in #4210).
+
+**Six executor-derived Zod contracts.** `GetRecordConfigSchema`,
+`CreateRecordConfigSchema`, `UpdateRecordConfigSchema`,
+`DeleteRecordConfigSchema`, `ScreenConfigSchema` (+ `ScreenFieldConfigSchema`)
+and `MapConfigSchema` in `automation/builtin-node-config.zod.ts`, each written
+by reading the executor rather than transcribing the descriptor literal, so the
+new bidirectional ledger test is evidence rather than a tautology. Contract
+exports only — nothing parses with them yet (#4045 step 3b, gated on the #4059
+warning data).
+
+**Seven capabilities the executors honour are now authorable.** Each was read
+by the executor and offered by no form — online or offline — so it was reachable
+only by hand-written metadata:
+
+- `get_record.fields` — the query projection, passed straight into
+  `find`/`findOne`;
+- `screen.recordId` — the record `mode: 'edit'` opens; the form declared the
+  edit mode while offering no way to name its target;
+- `screen.fields[].options` / `defaultValue` / `placeholder` — all three
+  forwarded into the ScreenSpec the client renders, so a select field's choices
+  could not be authored in Studio at all. Same nested repeater position as the
+  `visibleWhen` gap #3528 was filed for;
+- `map.indexVariable` and `map.input` — the index binding and the per-item
+  subflow params.
+
+**`map`'s undeclared `flow` alias graduates to the conversion layer.** The
+executor carried `cfg.flowName ?? cfg.flow` for a spelling no schema ever
+described — the `notify.source` shape (Prime Directive #12). The bare fallback
+is deleted and `flow-node-map-flow-alias` (protocol 17, retires at 18) renames
+it at load, including the `AutomationEngine.registerFlow` rehydration seam.
+
+**`assignment` is pinned as deliberately un-reconcilable**, with the reason on
+record: with no `assignments` wrapper its top-level config keys ARE the author's
+variable names, so no fixed key set can describe it and a catchall Zod would
+reconcile vacuously. What the ledger pins instead is that the form offers
+exactly the canonical `assignments` map and that the map stays open.
+
+With this, every builtin that publishes a `configSchema` is reconciled against
+its executor, and the ones that publish none each have a recorded reason.

.changeset/empty-capability-answers-501.md

@@ -0,0 +1,22 @@
+---
+'@objectstack/runtime': major
+---
+
+A dispatcher domain whose route is mounted but whose implementation is absent answers **501**, not a 404 that blames the route (#4093 follow-up).
+
+Two different facts were being answered with whatever each domain happened to reach for, and only `mcp` told them apart:
+
+- **The route is not there.** `/mcp` when the server is disabled for the environment; `/analytics` when the service is unserveable, because `dispatcher-plugin` gates the *mount* and never registers those paths (#4000). A path the server does not expose is a **404** from the host's own router. **Unchanged** — that half was already right.
+- **The route is there; the implementation is not.** Every unconditionally-mounted domain. The request reached a handler that had nothing to delegate to. That is **501**, and it is what changes here.
+
+`/automation` and `/notifications` returned `{ handled: false }`, which looks neutral but lands on the dispatcher plugin's single exit: `404 ROUTE_NOT_FOUND` with the hint *"No handler matched this request. Check the API discovery endpoint for available routes."* Both halves were false — a handler did match, and discovery correctly does not list the route, so the hint pointed at a page that would never mention it. An operator reads that as a routing bug and goes looking for one that does not exist.
+
+`/ui` answered **503**, which claims the condition is temporary. An uninstalled MetadataPlugin does not become installed by retrying.
+
+`/ai` answered **404** for the same mounted-but-unimplemented case.
+
+The refusal now carries the same remedy sentence discovery reports for that slot (`serviceUnavailableMessage`, shared via `@objectstack/spec/system`), so the wall and the discovery entry cannot drift into naming different fixes — `POST /api/v1/automation` on a stack without the service answers `501 Install @objectstack/service-automation to enable`. `/ai` keeps a local message: its real provider ships outside this workspace as a Cloud/EE package, so the shared table — verified against workspace packages — records no entry and would otherwise describe it as "nothing ships".
+
+Deliberately unchanged: `analytics`'s route-mount gate (a genuinely absent path); `mcp`'s 404-vs-501 pair, which is the model; the `handled: false` at the END of each domain, which means "no sub-route matched" and is a true 404; `GET /ai/agents`'s empty-list 200, a deliberate courtesy for the console's per-navigation poll; and `/ai`'s `503 routes not yet initialized`, which is a different condition (service present, internal state unready) and may genuinely be transient.
+
+FROM → TO: requests to `/automation`, `/notifications`, `/ui/*` and `/ai/*` on a deployment lacking the backing service now get `501` (with the package to install) instead of `404`/`503`. Anything branching on the old status should branch on `status >= 400` or read the error code; the capability was equally unavailable before, so no working flow changes. Discovery is unaffected — it already reported these slots `unavailable` and advertised no route for them.

.changeset/export-honors-search-term.md

@@ -0,0 +1,34 @@
+---
+"@objectstack/rest": patch
+---
+
+fix(rest): `GET /data/:object/export` honours a `search` term
+
+The streaming export route accepted `filter` and `orderby` but had no way to
+carry the term a user had typed into the list's search box. So exporting after
+a search downloaded the **unsearched superset** — more rows than the screen
+showed, in a file that looks authoritative, with nothing indicating the
+difference. The route's own comment claimed the opposite: that it "mirrors the
+active view's filter + sort so the exported file matches what the user sees".
+
+Same family as a dropped filter (objectstack#3948, objectstack#4181): a
+plausible answer that is quietly broader than the one asked for.
+
+Two new query params, both matching the list endpoint's semantics:
+
+- `search=<term>` — folded into `findData` as `$search`, so it **composes**
+  with `filter` (`{ $and: [filter, search] }`) rather than replacing it. Empty
+  or whitespace-only terms are ignored rather than applied as a blank predicate.
+- `searchFields=a,b` — the ADR-0061 override for which fields the term scans.
+  Only meaningful alongside `search`, and intersected with the object's allowed
+  searchable set by the engine, exactly as on the list endpoint.
+
+Unknown query params on this route were already ignored, so a client that sends
+`search` to an older server gets today's behaviour rather than an error.
+
+Covered by `export-integration.test.ts` against the real engine + protocol: the
+composition case is built so each half alone returns a different non-empty
+result and only "both applied" returns none. Reverting the route change fails 4
+of the tests. The file's in-memory driver also learned `$or` / `$contains` —
+without them a search predicate is a silent no-op and an "it filtered"
+assertion would pass for the wrong reason.

.changeset/lifecycle-event-registry-enforced.md

@@ -0,0 +1,64 @@
+---
+"@objectstack/spec": major
+"@objectstack/core": patch
+---
+
+fix(spec)!: retire the never-built typed-event system; the lifecycle registry now lists the events that actually fire (#4212 follow-up)
+
+The lifecycle-event surface promised a typed-event system that was never
+built, in three layers. `kernel/plugin-lifecycle-events.zod.ts` shipped ten
+payload schemas (`PluginRegisteredEvent`, `PluginErrorEvent`,
+`HookTriggeredEvent`, `KernelReadyEvent`, …) and a 21-name
+`PluginLifecycleEventType` enum — zero consumers for every export, and the
+enum was wrong in both directions: 17 names nothing fires, 10 real events
+missing. `contracts/plugin-lifecycle-events.ts` declared the same 17 dead
+names in `IPluginLifecycleEvents` next to 5 real ones, plus an
+`ITypedEventEmitter` interface nothing implements. All of it read as a
+promise; anyone who coded against it (hooking `plugin:started`, awaiting
+`plugin:error`) registered a handler that could never fire, with no error
+saying so — the same silent-drop shape as the #4212 lifecycle-hook family.
+
+Removed, with zero consumers verified repo-wide:
+
+- `kernel/plugin-lifecycle-events.zod.ts` and every export: `EventPhase`,
+  `PluginEventBase`, `PluginRegisteredEvent`, `PluginLifecyclePhaseEvent`,
+  `PluginErrorEvent`, `ServiceRegisteredEvent`, `ServiceUnregisteredEvent`,
+  `HookRegisteredEvent`, `HookTriggeredEvent`, `KernelEventBase`,
+  `KernelReadyEvent`, `KernelShutdownEvent`, `PluginLifecycleEventType`
+  (schemas and inferred types).
+- `ITypedEventEmitter` from `contracts/plugin-lifecycle-events.ts`.
+- The 17 never-fired names from `IPluginLifecycleEvents`.
+
+`IPluginLifecycleEvents` is now the registry of the **14 events with a real
+emitter** — `kernel:{ready,bootstrapped,listening,shutdown}`, `app:seeded`,
+`metadata:reloaded` (payload `metadata` now optional, matching the documented
+contract), `external.schema.drift`, `ai:routes`, `auth:configure`, and the
+`{service}:ready` convention family (`mcp`, `automation`, `analytics`,
+`external-datasource`, `datasource-admin`) — each payload as observed at its
+fire site. A new `LifecycleEventName` union types
+`PluginContext.hook`/`trigger` in `@objectstack/core` as
+`LifecycleEventName | (string & {})`: known names autocomplete, custom
+cross-plugin names stay legal, existing callers compile unchanged. A pinning
+test asserts two-way equality between the interface keys and the fire-site
+inventory.
+
+FROM → TO:
+
+- `PluginLifecycleEventType` → `LifecycleEventName` (the union of names that
+  fire). There is no runtime enum; the bus is open by design.
+- Event payload schemas (`KernelReadyEvent`, `PluginErrorEvent`, …) → the
+  payload tuples on `IPluginLifecycleEvents`. No wire format existed or
+  exists; payloads are in-process arguments.
+- `ITypedEventEmitter` → `PluginContext.hook`/`trigger` (the emitter that
+  actually exists).
+- Handlers for the 17 dead names → delete them; they never ran. For plugin
+  phase observation use the boot report (ADR-0084); for per-plugin errors the
+  kernel throws/logs at the failing phase.
+
+Plain deletion rather than `retiredKey()` tombstones, per the #4233
+precedent: these keys were never authorable — they described runtime event
+payload records no config author can write, so the silent-strip class the
+authorable-surface ratchet guards against is vacuous. Its baseline entries
+and the `json-schema.manifest.json` keys are dropped deliberately in this PR.
+No ADR-0087 conversion: no stack metadata names these types; there is nothing
+for `os migrate meta` to rewrite.