fix(spec,rest,runtime): request bodies are checked against the schemas the catalog declares (#3899)

.changeset/request-body-schema-validation.md

@@ -0,0 +1,31 @@
+---
+'@objectstack/spec': patch
+'@objectstack/rest': minor
+'@objectstack/runtime': minor
+---
+
+Request bodies are now checked against the schemas the API catalog declares for them (#3899, the request-side dual of #3877).
+
+**Routes that now answer `400 VALIDATION_FAILED` + `fields[]` for a body violating their declared `requestSchema`** (previously the body was consumed raw, and a malformed one silently executed different semantics):
+
+- `POST /data/:object/query` — body must be a QueryAST (`FindDataRequestSchema`); a garbage body used to degrade into an unfiltered full read. The path `object` is now pinned into the forwarded query (a body `object` can no longer contradict the path).
+- `POST /data/:object` / `PATCH /data/:object/:id` — body must be a record object (`CreateDataRequestSchema` / `UpdateDataRequestSchema`).
+- `POST /data/:object/batch` — body must be a `BatchUpdateRequestSchema` (`operation` + `records[]`).
+- `POST /data/:object/createMany` — body must be a bare JSON array of records (`CreateManyDataRequestSchema`); `{ records: [...] }` (updateMany's envelope) is rejected with a pointer.
+- `POST /notifications/read` — body must be `{ ids: string[] }` (`MarkNotificationsReadRequestSchema`); a misnamed key used to become `markRead(userId, [])` — a 200 no-op that never cleared the badge.
+
+**Dispatcher automation routes now validate their bodies** (no catalog schema; hand-written guards):
+
+- `POST /automation` and `PUT /automation/:name` require a flow-definition object, and POST requires a non-empty `name` — a mistyped `name` used to register the flow under the key `undefined` and echo 200.
+- `POST /automation/:name/toggle` is strictly `{ enabled?: boolean }` — `{"enable": false}` (one letter off) used to ENABLE the flow and answer 200 `{enabled: true}`; it is now a 400 naming the offending key. An empty body still means enable.
+
+**`QuerySchema` now declares the search contract ADR-0061 actually serves** (additive): `search` accepts the canonical bare query string as well as the structured `FullTextSearch` form, and the server-validated `searchFields` narrowing is formally declared. Previously the schema declared only the object form while every surface (and the ADR's own conformance proof) sent the string — drift that surfaced the moment request bodies started being validated.
+
+**Catalog corrections in `@objectstack/spec` (`plugin-rest-api.zod.ts`)** — documentation-only tables:
+
+- `DEFAULT_NOTIFICATION_ROUTES` drops the four device/preferences endpoints — those server routes were removed in #3612 (never built), yet the table kept declaring them, `requestSchema` and all.
+- `DEFAULT_AUTOMATION_ROUTES`' trigger endpoint path is corrected `/trigger` → `/trigger/:name` (the mounted path; the flow name rides the path) and its `AutomationTriggerRequestSchema` declaration is removed — that schema never described this route's wire shape.
+- `DEFAULT_DATA_CRUD_ROUTES` gains the `POST /:object/query` entry (mounted since forever, previously undeclared), repoints create/update to the schemas the routes actually validate (`CreateDataRequestSchema` / `UpdateDataRequestSchema` — the old `CreateRequestSchema`/`UpdateRequestSchema` names described a `{ data }` envelope the wire never had), and drops `requestSchema` from GET/DELETE entries (path/query-bound inputs; nothing can violate them as a body).
+- New gates: catalog `requestSchema`/`responseSchema` strings must resolve to real exported Zod schemas, `requestSchema` may only sit on body-carrying methods, and every declared `requestSchema` on a mounted route has a violating-body → 400 conformance case (`packages/rest` + `packages/runtime` request-schema-gate suites).
+
+Migration: clients that already send the documented shapes are unaffected. If you relied on a malformed body being silently accepted (e.g. posting `{ records: [...] }` to `createMany`, a non-boolean `enabled` to toggle, or an off-schema analytics/query body), fix the request to the declared shape — the 400's `fields[]` names each offending key.

content/docs/data-modeling/queries.mdx

@@ -432,6 +432,21 @@ dotted `fields` path (`'customer.name'`) for a single related column:
 
 ## Full-Text Search
 
+`search` takes the query text; the server resolves which fields to search from object
+metadata (ADR-0061). Pass `searchFields` to narrow that set — it is intersected with what
+the object allows, so it can only narrow, never widen.
+
+```typescript
+{
+  object: 'article',
+  search: 'kubernetes deployment',
+  searchFields: ['title', 'body', 'tags']   // optional
+}
+```
+
+The structured form is equivalent and carries the experimental knobs below — `query` and
+`fields` mean exactly what `search` and `searchFields` do:
+
 ```typescript
 {
   object: 'article',
@@ -444,8 +459,8 @@ dotted `fields` path (`'customer.name'`) for a single related column:
 
 | Property | Type | Description |
 |:---|:---|:---|
-| `query` | `string` | Search text |
-| `fields` | `string[]` | Fields to search (optional — defaults to all searchable) |
+| `query` | `string` | Search text (the bare `search: '…'` string form) |
+| `fields` | `string[]` | Fields to search (optional — defaults to all searchable; the top-level spelling is `searchFields`) |
 | `fuzzy` | `boolean` | `[EXPERIMENTAL — not enforced]` Fuzzy matching for typo tolerance |
 | `operator` | `'and' \| 'or'` | `[EXPERIMENTAL — not enforced]` How to combine search terms |
 | `boost` | `Record<string, number>` | `[EXPERIMENTAL — not enforced]` Field relevance weights |

content/docs/protocol/objectql/query-syntax.mdx

@@ -64,7 +64,8 @@ interface QueryAST {
   object: string;                          // Target object (required)
   fields?: FieldNode[];                    // Projection (SELECT) — field names
   where?: FilterCondition;                 // Filtering (WHERE) — MongoDB-style $op
-  search?: FullTextSearch;                 // Full-text search ($search)
+  search?: string | FullTextSearch;        // Full-text search — the query text (canonical), or the structured form
+  searchFields?: string[];                 // Narrow the search (server-intersected — narrows only, never widens)
   orderBy?: SortNode[];                    // Ordering (ORDER BY)
   limit?: number;                          // Max records (LIMIT)
   offset?: number;                         // Skip records (OFFSET)
@@ -751,25 +752,36 @@ an `$or` of `$contains` predicates across the object's server-resolved searchabl
 already runs `$or`/`$contains`, so no driver support is needed (`SqlDriver` reports
 `supports.fullTextSearch: false`).
 
+`search` takes the **query text itself** — that is the canonical spelling (ADR-0061 D1:
+the client says *what* to search for, the server decides *which fields*), and it is what
+every surface sends. The structured form is equivalent for the two members that drive
+the expansion, and carries the experimental knobs below:
+
 ```typescript
+// Canonical — the server resolves the fields from object metadata
 const query: QueryAST = {
   object: 'article',
-  search: {
-    query: 'ObjectStack tutorial',
-    fields: ['title', 'content'],
-  },
+  search: 'ObjectStack tutorial',
+  searchFields: ['title', 'content'],   // optional narrowing
+  limit: 10,
+};
+
+// Structured form — `query` + `fields` mean exactly the same thing
+const structured: QueryAST = {
+  object: 'article',
+  search: { query: 'ObjectStack tutorial', fields: ['title', 'content'] },
   limit: 10,
 };
 ```
 
-Field resolution is server-side and never client-trusted: `search.fields` is
-**intersected** with the object's declared `searchableFields` (or, absent those, an
-auto-default of the name field plus short-text/enum fields), so naming a field outside
-that set can never widen the search — and over the REST/protocol ingress it is
-`400 INVALID_FIELD` outright (#4254), because the engine-side intersection alone used to
-drop the unknown name and fall back to scanning the full searchable set. Internal
-callers reaching `engine.find()` directly keep the tolerant intersection.
-Multiple whitespace-separated terms are AND-ed and
+Field resolution is server-side and never client-trusted: the requested fields
+(`searchFields`, or `search.fields` in the structured form) are **intersected** with the
+object's declared `searchableFields` (or, absent those, an auto-default of the name field
+plus short-text/enum fields), so naming a field outside that set can never widen the
+search — and over the REST/protocol ingress it is `400 INVALID_FIELD` outright (#4254),
+because the engine-side intersection alone used to drop the unknown name and fall back to
+scanning the full searchable set. Internal callers reaching `engine.find()` directly keep
+the tolerant intersection. Multiple whitespace-separated terms are AND-ed and
 fields are OR-ed. Case sensitivity is the **driver's**, not the expansion's: the
 expansion emits a plain `$contains`, which `SqlDriver` compiles to a parameterised
 `LIKE '%…%'` with no case folding — so the dialect's own `LIKE`/collation rules decide —

content/docs/references/api/contract.mdx

@@ -149,7 +149,8 @@ const result = ApiError.parse(data);
 | **object** | `string` | ✅ | Object name (e.g. account) |
 | **fields** | `string[]` | optional | Fields to retrieve — field names, optionally dotted to reach through a relationship (`owner.name`). Related *records* are selected with `expand`, not from inside this list. |
 | **where** | `any` | optional | Filtering criteria (WHERE) |
-| **search** | `{ query: string; fields?: string[]; fuzzy: boolean; operator: Enum<'and' \| 'or'>; … }` | optional | Full-text search configuration ($search parameter) |
+| **search** | `string \| { query: string; fields?: string[]; fuzzy: boolean; operator: Enum<'and' \| 'or'>; … }` | optional | Full-text search — the query text (canonical, ADR-0061 D1), or a structured FullTextSearch configuration |
+| **searchFields** | `string[]` | optional | Narrow the search to these fields (server-intersected with the allowed searchable set — can only narrow, never widen; ADR-0061 D1) |
 | **orderBy** | `{ field: string; order: Enum<'asc' \| 'desc'> }[]` | optional | Sorting instructions (ORDER BY) |
 | **limit** | `number` | optional | Max records to return (LIMIT) |
 | **offset** | `number` | optional | Records to skip (OFFSET) |
@@ -161,7 +162,7 @@ const result = ApiError.parse(data);
 | **having** | `any` | optional | HAVING — filter over the AGGREGATED rows (aggregation aliases + groupBy projections); applied engine-side after aggregation |
 | **windowFunctions** | `any` | optional | [REMOVED] `query.windowFunctions` was removed in @objectstack/spec 18 (#4286, ADR-0049) — `find()` never applied it: no engine or driver read the key on the query path, so every OVER clause it declared was silently dropped. Delete the key. Window functions are a SQL-driver capability behind `SqlDriver.findWithWindowFunctions(object, query)` (embedder-level; not on the `IDataDriver` contract or the REST surface); request-level analytics are `aggregations` + `groupBy`. |
 | **distinct** | `any` | optional | [REMOVED] `query.distinct` was removed in @objectstack/spec 18 (#4286, ADR-0049 / ADR-0078) — no driver ever rendered SELECT DISTINCT; the flag's only observable effect was MIS-WIRED: the REST list path treated a distinct query as not countable and silently degraded `total`/`hasMore` to a page-local estimate while still returning duplicate rows. Delete the key; `QueryBuilder.distinct()` was removed with it, and the count suppression is gone (`total` is truthful again). For unique values of one column use the SQL/memory drivers' `distinct(object, field)` door; for unique combinations, `groupBy`; for a deduplicated count, the `count_distinct` aggregation. |
-| **expand** | `Record<string, { object: string; fields?: string[]; where?: any; search?: object; … }>` | optional | Recursive relation loading map. Keys are lookup/master_detail field names; values are nested QueryAST objects that control select (`fields`) and filter (`where`, AND-merged with the batch $in), plus further expansion on the related object. The engine resolves expand via batch $in queries (driver-agnostic) with a default max depth of 3; per-parent `limit`/`offset`/`orderBy` are NOT applied on this path. |
+| **expand** | `Record<string, { object: string; fields?: string[]; where?: any; search?: string \| { query: string; fields?: string[]; fuzzy: boolean; operator: Enum<'and' \| 'or'>; … }; … }>` | optional | Recursive relation loading map. Keys are lookup/master_detail field names; values are nested QueryAST objects that control select (`fields`) and filter (`where`, AND-merged with the batch $in), plus further expansion on the related object. The engine resolves expand via batch $in queries (driver-agnostic) with a default max depth of 3; per-parent `limit`/`offset`/`orderBy` are NOT applied on this path. |
 
 
 ---

content/docs/references/api/protocol.mdx

@@ -529,7 +529,7 @@ const result = AiAgentCapabilities.parse(data);
 | Property | Type | Required | Description |
 | :--- | :--- | :--- | :--- |
 | **object** | `string` | ✅ | The unique machine name of the object to query (e.g. "account"). |
-| **query** | `{ object: string; fields?: string[]; where?: any; search?: object; … }` | optional | Structured query definition (filter, sort, select, pagination). |
+| **query** | `{ object: string; fields?: string[]; where?: any; search?: string \| { query: string; fields?: string[]; fuzzy: boolean; operator: Enum<'and' \| 'or'>; … }; … }` | optional | Structured query definition (filter, sort, select, pagination). |
 
 
 ---

content/docs/references/data/data-engine.mdx

@@ -548,7 +548,7 @@ QueryAST-aligned query options for IDataEngine.find() operations
 | **top** | `number` | optional |  |
 | **cursor** | `any` | optional | [REMOVED] `query.cursor` was removed in @objectstack/spec 18 (#4286, ADR-0049) — no driver ever implemented keyset pagination, so the cursor was accepted and ignored and every page came back identical (a caller looping "until hasMore is false" never terminates). Delete the key; `QueryBuilder.cursor()` was removed with it. Express the keyset as an ordinary `where` predicate on your sort key — `where: { created_at: { $gt: last.created_at } }` with the matching `orderBy` — which every driver executes with canonicalised comparands. A first-class cursor, if ever built, will be a response-minted opaque token, not this caller-built record. |
 | **search** | `{ query: string; fields?: string[]; fuzzy: boolean; operator: Enum<'and' \| 'or'>; … }` | optional |  |
-| **expand** | `Record<string, { object: string; fields?: string[]; where?: any; search?: object; … }>` | optional |  |
+| **expand** | `Record<string, { object: string; fields?: string[]; where?: any; search?: string \| { query: string; fields?: string[]; fuzzy: boolean; operator: Enum<'and' \| 'or'>; … }; … }>` | optional |  |
 | **distinct** | `any` | optional | [REMOVED] `query.distinct` was removed in @objectstack/spec 18 (#4286, ADR-0049 / ADR-0078) — no driver ever rendered SELECT DISTINCT; the flag's only observable effect was MIS-WIRED: the REST list path treated a distinct query as not countable and silently degraded `total`/`hasMore` to a page-local estimate while still returning duplicate rows. Delete the key; `QueryBuilder.distinct()` was removed with it, and the count suppression is gone (`total` is truthful again). For unique values of one column use the SQL/memory drivers' `distinct(object, field)` door; for unique combinations, `groupBy`; for a deduplicated count, the `count_distinct` aggregation. |
 
 

content/docs/references/data/mapping.mdx

@@ -52,7 +52,7 @@ const result = FieldMapping.parse(data);
 | **fieldMapping** | `{ source: string \| string[]; target: string \| string[]; transform: Enum<'none' \| 'constant' \| 'lookup' \| 'split' \| 'join' \| 'javascript' \| 'map'>; params?: object }[]` | ✅ |  |
 | **mode** | `Enum<'insert' \| 'update' \| 'upsert'>` | ✅ |  |
 | **upsertKey** | `string[]` | optional | Fields to match for upsert (e.g. email) |
-| **extractQuery** | `{ object: string; fields?: string[]; where?: any; search?: object; … }` | optional | Query to run for export only |
+| **extractQuery** | `{ object: string; fields?: string[]; where?: any; search?: string \| { query: string; fields?: string[]; fuzzy: boolean; operator: Enum<'and' \| 'or'>; … }; … }` | optional | Query to run for export only |
 | **errorPolicy** | `Enum<'skip' \| 'abort' \| 'retry'>` | ✅ |  |
 | **batchSize** | `number` | ✅ |  |
 

content/docs/references/data/query.mdx

@@ -126,7 +126,8 @@ Type: `string`
 | **object** | `string` | ✅ | Object name (e.g. account) |
 | **fields** | `string[]` | optional | Fields to retrieve — field names, optionally dotted to reach through a relationship (`owner.name`). Related *records* are selected with `expand`, not from inside this list. |
 | **where** | `any` | optional | Filtering criteria (WHERE) |
-| **search** | `{ query: string; fields?: string[]; fuzzy: boolean; operator: Enum<'and' \| 'or'>; … }` | optional | Full-text search configuration ($search parameter) |
+| **search** | `string \| { query: string; fields?: string[]; fuzzy: boolean; operator: Enum<'and' \| 'or'>; … }` | optional | Full-text search — the query text (canonical, ADR-0061 D1), or a structured FullTextSearch configuration |
+| **searchFields** | `string[]` | optional | Narrow the search to these fields (server-intersected with the allowed searchable set — can only narrow, never widen; ADR-0061 D1) |
 | **orderBy** | `{ field: string; order: Enum<'asc' \| 'desc'> }[]` | optional | Sorting instructions (ORDER BY) |
 | **limit** | `number` | optional | Max records to return (LIMIT) |
 | **offset** | `number` | optional | Records to skip (OFFSET) |

packages/metadata-protocol/src/protocol.ts

@@ -840,10 +840,10 @@ function mergeDroppedFieldEvents(events: DroppedFieldsEvent[]): DroppedFieldsEve
  * which is a behavior change to call out in that changeset.
  */
 const QUERY_AST_KEYS: Readonly<Record<keyof QueryAST, true>> = {
-    object: true, fields: true, where: true, search: true, orderBy: true,
-    limit: true, offset: true, top: true, cursor: true, joins: true,
-    aggregations: true, groupBy: true, having: true, windowFunctions: true,
-    distinct: true, expand: true,
+    object: true, fields: true, where: true, search: true, searchFields: true,
+    orderBy: true, limit: true, offset: true, top: true, cursor: true,
+    joins: true, aggregations: true, groupBy: true, having: true,
+    windowFunctions: true, distinct: true, expand: true,
 };
 
 /**
@@ -882,7 +882,10 @@ const RESERVED_LIST_QUERY_PARAMS: ReadonlySet<string> = new Set([
     ...Object.keys(QUERY_AST_KEYS),
     // Transport-only extras the normalizer consumes but the AST does not name.
     'count',        // ?count / $count — response flag, not a projection
-    'searchFields', // ?searchFields / $searchFields — ADR-0061 override
+    // `searchFields` used to be listed here as such an extra. It is a named
+    // AST key since #3899 declared it (ADR-0061 P1), so it now arrives through
+    // the spread above and the type-level pin covers it — the hand-maintained
+    // copy would have been a second source that could silently fall out of step.
     // Server-derived, never caller input (stripped then re-set from `request`).
     'context',
 ]);

packages/plugins/driver-mongodb/src/mongodb-datetime-storage.test.ts

@@ -20,18 +20,11 @@
  */
 
 import { describe, it, expect, beforeAll, afterAll, beforeEach } from 'vitest';
-import { MongoMemoryServer } from 'mongodb-memory-server';
+import type { MongoMemoryServer } from 'mongodb-memory-server';
 import { MongoDBDriver } from './mongodb-driver.js';
+import { createTestMongod } from './test-mongod.js';
 
-let sharedMongod: MongoMemoryServer | undefined;
-try {
-  sharedMongod = await MongoMemoryServer.create({ instance: { launchTimeout: 60_000 } });
-} catch (err) {
-  console.warn(
-    '[driver-mongodb] Skipping datetime-storage suite — mongodb-memory-server could not start: ' +
-      `${(err as Error)?.message ?? String(err)}`,
-  );
-}
+const sharedMongod: MongoMemoryServer | undefined = await createTestMongod('datetime-storage');
 
 const ids = (rows: any[]) => rows.map((r: any) => r.id).sort();