Merge origin/main into claude/datasource-admin-error-code-y1eels

Resolves the #3973/#4229 overlap: the per-module sendOk/sendError pairs this
branch had annotated were consolidated into @objectstack/types on main, so the
local definitions go and the #4249 notes move to the module docs. With the
shared sendError typing `code` as the closed ErrorCode union, SERVICE_ERROR_CODE
is now Record<ServiceName, ErrorCode> — the mis-attribution class this branch
fixes is a compile error going forward.

Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_01Di9hefxJMFD3rxiP2G2Eug

Author: claude

.changeset/adr-0104-advertise-open-gates.md

@@ -0,0 +1,34 @@
+---
+"@objectstack/objectql": patch
+"@objectstack/cli": patch
+---
+
+feat(migrate,objectql): the upgrade path names the data migrations that are still open here (#3438, ADR-0104 2026-07-30)
+
+Both value-shape gates fail toward leniency: a deployment that never runs its
+migration keeps warning instead of rejecting, and keeps every released file
+forever. That default is right — and completely silent, so the gate could sit
+open for the life of a deployment without anyone learning that one command ends
+it. A gate nobody is told about is served by nobody.
+
+Two announcements, each where an upgrade actually looks:
+
+- **`os migrate meta --from 16`** now ends by naming the data migrations a
+  chain crossing into 17 leaves behind — `files-to-references`, `value-shapes`
+  — with what each unlocks, scoped to the field classes the author's own
+  metadata declares (an app with no media field is never told about the file
+  migration). `--json` carries the same list as `dataMigrations`. The command
+  reads no database, so it reports what remains *to do*, never what a given
+  deployment has *done*.
+- **The server logs one line per open gate at boot**, naming the command that
+  closes it. Only the lax posture announces itself — a verified gate already
+  logs that it is enforcing, and an app declaring neither class of field costs
+  nothing and says nothing. This is the half that can speak to a deployment's
+  actual data, because it is the half with the database.
+
+Nothing about enforcement changes: same gates, same flags, same fail-toward-
+leniency default. The advisory runs on `kernel:bootstrapped` rather than
+`kernel:ready`, deliberately — the answer depends on the storage service's own
+ready handler, which registers `sys_migration` and may attest a store it just
+created, and racing it would tell a brand-new deployment its gates are open
+moments after they closed.

.changeset/published-files-whitelist.md

@@ -0,0 +1,57 @@
+---
+"@objectstack/formula": patch
+"@objectstack/sdui-parser": patch
+"@objectstack/connector-mcp": patch
+"@objectstack/connector-openapi": patch
+"@objectstack/connector-rest": patch
+"@objectstack/connector-slack": patch
+"@objectstack/embedder-openai": patch
+"@objectstack/knowledge-memory": patch
+"@objectstack/knowledge-ragflow": patch
+"@objectstack/plugin-approvals": patch
+"@objectstack/plugin-email": patch
+"@objectstack/plugin-pinyin-search": patch
+"@objectstack/plugin-reports": patch
+"@objectstack/plugin-sharing": patch
+"@objectstack/plugin-webhooks": patch
+"@objectstack/service-cluster": patch
+"@objectstack/service-cluster-redis": patch
+"@objectstack/service-datasource": patch
+"@objectstack/service-sms": patch
+"@objectstack/trigger-api": patch
+---
+
+chore(packaging): 20 packages stop publishing their sources, tests and build tooling (#4248)
+
+These 20 packages declared no `files` field, so npm fell back to packing the
+whole package directory. `npm pack --dry-run` on `@objectstack/plugin-webhooks`
+listed **21 files** — 15 under `src/`, three of them unit tests
+(`auto-enqueuer.test.ts`, `bootstrap-declared-webhooks.test.ts`, …), plus the
+build-time `scripts/i18n-extract.config.ts`. `dist/` lands on top of that at
+publish time rather than instead of it, so consumers were installing the
+TypeScript sources and the test suite alongside the artifact they asked for.
+
+Each now declares `"files": ["dist", "README.md"]`, matching the 29 packages
+that already did. Nothing a consumer imports moves: every `main` / `types` /
+`exports` target in all 20 already resolved inside `dist/`, which the new
+`check:published-files` guard verifies rather than assumes. The visible change
+is a smaller install and a smaller dependency-scanning surface — `npm pack` on
+`@objectstack/plugin-webhooks` now yields 2 files plus `dist/`.
+
+The other half of the fix is the gate. Half the packages declaring `files` and
+half not was the #3786 shape — a hand-copied convention with nothing enforcing
+it, where whoever forgets the line gets no signal at all. `check:published-files`
+(new, wired into the always-required `lint` job) holds every non-private
+workspace package to four invariants: `files` is **declared**; it is
+**sufficient** (covers every entry point, so tightening a whitelist cannot ship
+a package that fails to resolve); it is **minimal** (admits no test, test-harness
+config or build script); and anything beyond `dist` + `README.md` is
+**registered** with a reason, reconciled in both directions so a stale exemption
+is an error rather than dead text. `@objectstack/spec` is the one package with
+registered extras — its `.zod.ts` sources, JSON Schemas, liveness ledgers and
+`CHANGELOG.md` are product, not build input.
+
+This also closes an assumption #4206 was resting on. Excluding `<pkg>/scripts/**`
+from the docs-drift implementation test is sound only while no package publishes
+`scripts/` as runtime code; that held, but it held because someone read all three
+offenders by hand. It is now checked on every PR.

.changeset/query-field-node-object-form-removed.md

@@ -0,0 +1,68 @@
+---
+"@objectstack/spec": major
+"@objectstack/objectql": patch
+"@objectstack/metadata-protocol": patch
+"@objectstack/driver-mongodb": patch
+"@objectstack/driver-sql": patch
+"@objectstack/driver-memory": patch
+---
+
+refactor(data)!: a select-list entry is a field name — the nested-select object form is removed (#4196)
+
+`FieldNode` declared two forms for one entry of `QueryAST['fields']`:
+
+```ts
+type FieldNode =
+  | string                                                  // "name"
+  | { field: string; fields?: FieldNode[]; alias?: string }; // nested select
+```
+
+The object form was **declared-but-inert**. Nothing produced it, and nothing
+read `.fields` or `.alias` — every consumer on the path treats the list as
+`string[]`: `objectql`'s formula projection and its two known-field filters,
+`driver-sql`'s `select()`, `driver-memory`'s `projectFields`. `driver-mongodb`
+keyed its projection with the entry itself, so an object entry asked for a
+column literally named `"[object Object]"`, and the REST ingress stringified
+each entry before comparing it to the field map, so the same entry came back as
+`400 INVALID_FIELD: Unknown field '[object Object]'` — a rejection naming
+something the caller never wrote. An author who wrote
+`fields: [{ field: 'owner', fields: ['name'] }]` got it accepted by validation
+and then dropped or mangled, depending on the driver (ADR-0078 silently-inert
+declaration; ADR-0049 enforce-or-remove).
+
+The capability the object form described is already served, by a different key.
+Removing the second spelling rather than lowering it into the first is Prime
+Directive #12: one capability, one contract.
+
+**FROM → TO**
+
+| Was | Now |
+| :--- | :--- |
+| `fields: [{ field: 'owner', fields: ['name'] }]` | `expand: { owner: { object: 'user', fields: ['name'] } }` |
+| `fields: [{ field: 'owner' }]` | `fields: ['owner']` |
+| `fields: [{ field: 'owner', fields: ['name'] }]`, one column only | `fields: ['owner.name']` (dotted path) |
+| `fields: [{ field: 'total', alias: 't' }]` | `aggregations` / `windowFunctions` — they carry the live `alias` |
+
+The one-line fix: **a `fields[]` entry is a string.** Move nested selection to
+`expand`, which the engine resolves through batch `$in` queries (default max
+depth 3).
+
+There is no `os migrate meta` step, and deliberately so: `QueryAST` is a request
+shape, never stored in stack metadata, so the chain has no source to rewrite. It
+is registered as an ADR-0087 D3 **semantic** migration
+(`query-field-node-object-form-retired`) on the protocol-18 step instead — the
+`EnhancedApiError.fieldErrors` / `BatchOptions.validateOnly` precedent. Callers
+move their own select lists, and both channels tell them how:
+
+- **The parse.** `FieldNodeSchema` narrows to `z.string()` with an error map that
+  answers an object entry with the prescription above, not "expected string,
+  received object". `z.input` becomes `string`, so `tsc` fails at the authoring
+  site first.
+- **The ingress.** `assertProjectionFieldsExist` judges the entry's *shape*
+  before consulting the object's field map — it is wrong about the shape, not
+  about this object, and a registry-less host would otherwise pass it to a driver
+  that cannot read it. The 400 now names the retired form instead of the field
+  `"[object Object]"`.
+
+No runtime behaviour changes for anything that ever worked; the defensive
+unwrapping the drivers had grown against a shape nothing sends goes with it.

.changeset/shared-response-envelope-builders.md

@@ -0,0 +1,103 @@
+---
+"@objectstack/types": minor
+"@objectstack/rest": patch
+"@objectstack/service-storage": patch
+"@objectstack/service-settings": patch
+"@objectstack/service-datasource": patch
+"@objectstack/service-i18n": patch
+"@objectstack/plugin-sharing": patch
+---
+
+refactor(types,rest,services,plugin-sharing): one shared writer for the response envelope, and `error.code` is enforced at compile time (#3973)
+
+`BaseResponseSchema` declares one envelope for every REST body the platform
+emits. It declared it once; the code that *wrote* it was copied per route
+module. After #3843 and #3983 converted the last drifting one, seven modules
+each carried their own two-line `sendOk` / `sendError` pair — so the envelope's
+shape lived in fourteen places rather than one.
+
+`pnpm check:route-envelope` proved those seven copies agreed, which is why this
+is a cleanup rather than a bug fix. But a guard proves agreement; it does not
+create it. An eighth module starts by copying the pair again — not
+hypothetically: `share-link-routes.ts` was found already drifting by the
+repo-wide scan, and its drift had broken `client.shareLinks.create()` and
+`.list()` through `unwrapResponse` (#3983).
+
+## What moved
+
+`sendOk` / `sendError` now live once, in `@objectstack/types`
+(`response-envelope.ts`), and all seven modules import them:
+
+| Module |
+|---|
+| `service-storage/storage-routes.ts` |
+| `service-settings/settings-routes.ts` |
+| `service-datasource/admin-routes.ts` |
+| `rest/external-datasource-routes.ts` |
+| `rest/package-routes.ts` |
+| `service-i18n/i18n-service-plugin.ts` |
+| `plugin-sharing/share-link-routes.ts` |
+
+Placement was the open question in #3973, not design. `packages/spec` is
+schemas-only (Prime Directive #2), and the callers span `rest`, four
+`services/*` and one `plugins/*`, which rules out anything depending on them.
+`@objectstack/types` depends on nothing but `@objectstack/spec`, so every caller
+can reach it, and it is already where the repo puts a helper the HTTP boundaries
+share — `looksLikeInternalErrorLeak` (#3867) sits one file over and made the
+same argument first.
+
+The builders take a structural `{ status(n), json(body) }`, so the package
+imports no HTTP contract at all: `IHttpResponse` satisfies it, and so does the
+`any`-typed `res` the older modules carry.
+
+## `error.code` is now checked by the compiler
+
+All seven copies typed the parameter `code: string`. ADR-0112 (#3841) closed the
+vocabulary — `ErrorCode` is `StandardErrorCode ∪ ERROR_CODE_LEDGER` — but an
+invented code was still caught only at runtime, by a conformance suite parsing a
+driven body, i.e. only on routes some test happened to drive.
+
+The shared `sendError` types `code` as `ErrorCode`, so an unregistered code now
+fails to compile, at every call site at once:
+
+```ts
+sendError(res, 400, 'NOT_A_REGISTERED_CODE', 'invented');
+// Argument of type '"NOT_A_REGISTERED_CODE"' is not assignable to parameter of type 'ErrorCode'.
+```
+
+This cost no call-site churn: every code the seven modules emit was already
+registered.
+
+## `extra` is closed at the same place
+
+`sendError`'s last parameter is `Pick<ApiError, 'category' | 'httpStatus' |
+'details' | 'requestId'>` — exactly what `ApiErrorSchema` declares beside `code`
+and `message`.
+
+It was `Record<string, unknown>` while `settings-routes` still hung `namespace` /
+`key` / `reason` / `fields` beside `code`. Those bodies passed every gate anyway:
+`ApiErrorSchema` is a plain `z.object`, so unknown keys were STRIPPED rather than
+rejected, and `envelopeViolations` inspects only the body's top level —
+conformant *by stripping* rather than by declaration. #4224 moved that module
+onto `details`, which is what lets the parameter close here. Closing it at the
+shared builder is the part that lasts: an undeclared sibling is now a compile
+error in every module at once, rather than a key that quietly evaporates in
+whichever module reintroduces it.
+
+## Nothing changes on the wire
+
+The seven pairs were identical modulo the optional `status` and `extra`
+parameters this one unions, and each module's driven conformance suite still
+parses its real bodies against the real spec schemas. One internal call site was
+rewritten: `package-routes` passed `details` positionally and now passes
+`{ details }`, producing the same `error.details` it always did.
+
+## The guard got stronger
+
+`scripts/check-route-envelope.mjs` counts response write sites per module. A
+module that routes everything through the shared pair builds **none** itself, so
+the seven now declare `0 / 0 / 0` where they used to declare `2 / 1 / 1`, and the
+shared pair is pinned separately at `2 / 1 / 1` so the invariant stays total for
+the surface rather than per-module. What the count asserts is no longer "your two
+builders are the enveloped ones" but "you have no builders" — and a new route
+that hand-rolls a body still moves it off zero and fails.

.github/workflows/lint.yml

@@ -166,6 +166,22 @@ jobs:
       - name: Node-version drift guard
         run: pnpm check:node-version
 
+      # #4248 packaging-hygiene guard. Without a `files` whitelist npm packs the
+      # whole package directory, and 20 of the 49 publishable packages declared
+      # none — so consumers installed TypeScript sources, unit tests and build
+      # tooling, with dist/ landing on top of them rather than instead of them
+      # (@objectstack/plugin-webhooks: 21 files, three of them unit tests). The
+      # other 29 did declare it, so this was a hand-copied line with no gate —
+      # the #3786 shape, where whoever forgets it gets no signal at all. Also
+      # checks the whitelist is SUFFICIENT (covers every entry point, so
+      # tightening one cannot ship a package that fails to resolve) and MINIMAL
+      # (admits no test or build script), which keeps #4206's "`<pkg>/scripts/**`
+      # is never runtime code" assumption continuously verified instead of
+      # hand-checked. Runs its own --self-test first: the pattern semantics can
+      # be wrong while every package is right.
+      - name: Published-files whitelist guard
+        run: pnpm check:published-files
+
   typecheck:
     name: TypeScript Type Check
     runs-on: ubuntu-latest

content/docs/deployment/cli.mdx

@@ -745,6 +745,22 @@ them temporarily, `OS_ALLOW_LAX_MEDIA_VALUES=1` (media) and
 per class, and re-running the corresponding migration re-establishes its flag
 from the data itself.
 
+#### How you find out a gate is open
+
+You are told, in the two places an upgrade actually looks.
+
+**`os migrate meta --from 16`** — the metadata half of the upgrade — ends by
+naming the data migrations that remain, scoped to the field classes *your*
+metadata declares. It reads no database, so it reports what is left to do,
+never what this deployment has already done. The machine-readable output
+carries the same list under `dataMigrations`.
+
+**The server, once per boot**, logs one line per gate that is still open here
+and the command that closes it. Only the lax posture announces itself: a
+closed gate logs that it is enforcing, and an app that declares neither class
+of field says nothing at all. So a running deployment always tells you the
+state of its own data — which is the question `os migrate meta` cannot answer.
+
 ### Scaffolding
 
 | Command | Alias | Description |

content/docs/kernel/contracts/data-engine.mdx

@@ -85,7 +85,7 @@ Defined by `EngineQueryOptionsSchema` in `@objectstack/spec`:
 ```typescript
 interface EngineQueryOptions {
   where?: FilterCondition;                   // WHERE clause — MongoDB-style $op
-  fields?: FieldNode[];                      // SELECT fields
+  fields?: FieldNode[];                      // SELECT — field names ('name', 'owner.name')
   orderBy?: SortNode[];                      // ORDER BY
   limit?: number;                            // LIMIT
   offset?: number;                           // OFFSET
@@ -387,7 +387,7 @@ The following legacy parameter names are accepted by the RPC layer for backward
 | Legacy (Deprecated) | Canonical | Notes |
 |:---------------------|:----------|:------|
 | `filter` | `where` | FilterCondition object |
-| `select` | `fields` | Array of FieldNode |
+| `select` | `fields` | Array of FieldNode (field-name strings) |
 | `sort` | `orderBy` | Array of `{ field, order }` |
 | `skip` | `offset` | Number |
 | `populate` | `expand` | Record of field → QueryAST |

content/docs/protocol/objectql/query-syntax.mdx

@@ -62,7 +62,7 @@ import type { QueryAST } from '@objectstack/spec/data';
 // QueryAST — full structure
 interface QueryAST {
   object: string;                          // Target object (required)
-  fields?: FieldNode[];                    // Projection (SELECT)
+  fields?: FieldNode[];                    // Projection (SELECT) — field names
   where?: FilterCondition;                 // Filtering (WHERE) — MongoDB-style $op
   search?: FullTextSearch;                 // Full-text search ($search)
   orderBy?: SortNode[];                    // Ordering (ORDER BY)
@@ -97,7 +97,6 @@ on the `find()` path:
 | `cursor` | Accepted by `EngineQueryOptions`, but no driver implements keyset pagination |
 | `distinct` | Not applied by `find()`; the SQL and in-memory drivers expose a separate `distinct(object, field, filters?)` method instead |
 | `windowFunctions` | Not applied by `find()`; `SqlDriver.findWithWindowFunctions()` is a separate method that is not on the `IDataDriver` contract |
-| Object `FieldNode`s (`{ field, fields }`) | The engine drops non-string entries from `fields` — use `expand` to project related records |
 | `search.fuzzy` / `boost` / `operator` / `minScore` / `language` / `highlight` | Parsed by the schema but ignored: only `query` and `fields` drive the expansion |
 
 `top` is the exception that *is* honored — the engine normalises it to `limit`.
@@ -122,12 +121,16 @@ interface AggregationNode {
   filter?: FilterCondition; // FILTER WHERE clause — protocol only, never applied
 }
 
-// FieldNode — field selection
-type FieldNode = string | {
-  field: string;
-  fields?: FieldNode[];  // nested select — protocol only, dropped by the engine
-  alias?: string;
-};
+// FieldNode — one entry of the select list. A field name, optionally dotted to
+// reach through a relationship ('owner.name'). Related *records* come from
+// `expand`, not from inside this list.
+//
+// The `{ field, fields, alias }` nested-select member this union used to carry
+// was REMOVED in protocol 18 (#4196): nothing produced it and nothing read
+// `.fields`/`.alias`, so it was dropped by the SQL and memory drivers,
+// projected as a column named "[object Object]" by MongoDB, and refused as an
+// unknown field by the REST ingress. `expand` is the one spelling.
+type FieldNode = string;
 
 // GroupByNode — GROUP BY target
 type GroupByNode = string | {
@@ -980,7 +983,7 @@ Similarly, the following legacy field names should not be used in new code:
 | `aggregate` (object map) | `aggregations` (array) | Array of `AggregationNode` objects |
 | `expand` (string array) | `expand` (Record) | Map of field name → nested QueryAST |
 | `skip` | `offset` | Number |
-| `select` | `fields` | Array of FieldNode |
+| `select` | `fields` | Array of FieldNode (field-name strings) |
 | `populate` | `expand` | Map of field name → nested QueryAST |
 </Callout>