Skip to content
Merged
Show file tree
Hide file tree
Changes from all commits
Commits
File filter

Filter by extension

Filter by extension

Conversations
Failed to load comments.
Loading
Jump to
Jump to file
Failed to load files.
Loading
Diff view
Diff view
80 changes: 80 additions & 0 deletions .changeset/query-ast-inert-request-surface.md
Original file line number Diff line number Diff line change
@@ -0,0 +1,80 @@
---
"@objectstack/spec": major
"@objectstack/plugin-security": patch
---

refactor(data)!: the QueryAST request surface stops declaring what no executor runs — `joins` and `windowFunctions` removed, six search flags and `aggregations[].filter` marked experimental, and the liveness ledger now governs the query surface (#4286)

#4196 removed one declared-but-inert member from `FieldNode`. Applying the same
method to the rest of the request surface (#4286) found 12 more members of
`QueryAST` that no executor runs — `packages/objectql`'s `engine.ts` contains
zero reads of any of them on the query path. This change dispositions the
mechanical tiers and closes the gate that let the class stay invisible.

**Removed (tombstoned): `query.joins` and `query.windowFunctions`.**

- `joins` — no engine or driver ever read it; a query carrying it silently ran
as a single-table query. Related-record retrieval already has a live
spelling: `expand`. The orphaned `JoinNode` / `JoinNodeInput` /
`JoinNodeSchema` / `JoinType` / `JoinStrategy` exports are deleted with the
key (`data/JoinNode`, `data/JoinType`, `data/JoinStrategy` leave the
published JSON schemas).
- `windowFunctions` — `find()` never applied it, so every OVER clause it
declared was silently dropped. The one live door is the SQL driver's own
`findWithWindowFunctions(object, query)` (driver-level, not on the
`IDataDriver` contract), and its input is a flat driver shape the spec
vocabulary never matched — `WindowFunctionNodeSchema` declared `field` /
`over` / `frame` members that door never read. The `WindowFunction` /
`WindowSpec` / `WindowFunctionNode` exports are deleted with the key.

**FROM → TO**

| Was | Now |
| :--- | :--- |
| `joins: [{ type: 'inner', object: 'customer', on: … }]` | `expand: { customer_id: { object: 'customer', fields: ['name'] } }` |
| `joins` for one related column | `fields: ['customer_id.name']` (dotted path) |
| `windowFunctions: [{ function: 'rank', … }]` in a query | `aggregations` + `groupBy`, or rankings in report/dashboard metadata |
| OVER-clause SQL from an embedder | `sqlDriver.findWithWindowFunctions(object, { windowFunctions: [{ function, alias, partitionBy?, orderBy? }] })` |

The one-line fix: **delete the key**. Both are `retiredKey()` tombstones on the
non-strict `BaseQuerySchema`, so authoring either fails `tsc` (input type
`never`) and a query still carrying one — even as an empty array — fails to
parse with the prescription itself. `QueryAST` is a request shape, never stored
in stack metadata, so there is no `os migrate meta` step: the removals are
registered as protocol-18 **semantic** migrations (`query-joins-retired`,
`query-window-functions-retired`), the #4196 precedent.

Compat note for the REST boundary: both names remain **reserved** list-query
parameters while the tombstones live (`retiredKey()` keeps a key in
`keyof QueryAST`, which feeds `RESERVED_LIST_QUERY_PARAMS`), so nothing changes
for objects with fields named `joins`/`windowFunctions` — the un-reservation
happens when the tombstones age out, and is called out in
`metadata-protocol`'s `QUERY_AST_KEYS` comment for whoever does it.

**Marked `[EXPERIMENTAL — not enforced]` (no wire or compat impact):**
`search.fuzzy` / `operator` / `boost` / `minScore` / `language` / `highlight`
(the ADR-0061 expansion reads only `query` + `fields`) and
`AggregationNode.filter` (a SQL `FILTER (WHERE …)` affordance neither the SQL
builders nor the in-memory fallback applies). Authoring one is now a
declaration, not a silent no-op.

**Deliberately NOT dispositioned here** (they want a maintainer call, #4286
steps 3–4): `having` (the strongest enforce candidate — `engine.aggregate()`
currently rebuilds the driver AST without it), and `cursor` / `distinct`
(shipped SDK producers `QueryBuilder.cursor()` / `.distinct()`; `distinct` is
mis-wired — its only observable effect is suppressing the REST list count).
All three are recorded `dead` with evidence in the new ledger.

**The gate:** `QuerySchema` joins the liveness ledger through the gate's
`SPEC_ONLY_SCHEMAS` override (the `webhook` precedent) as governed type
`query` — the first governance of what *callers* write into a query rather
than what authors write into metadata files. `packages/spec/liveness/query.json`
classifies all 27 walked members (15 live with evidence, 7 experimental via
describe markers, 5 dead), so the next declared-but-inert request member fails
CI instead of needing a person to notice it.

`@objectstack/plugin-security` (patch): the FLS predicate guard's
`windowFunctions` walk is pruned — the clause no longer exists to leak through.
The `having` and `aggregations[].filter` walks stay, deliberately: those
members remain declared, and the guard being ready is what makes enforcing
them later safe.
115 changes: 40 additions & 75 deletions content/docs/data-modeling/queries.mdx
Original file line number Diff line number Diff line change
@@ -1,6 +1,6 @@
---
title: Query Syntax Cheat Sheet
description: One-page reference for ObjectStack QuerySchema — filters, sorts, pagination, aggregations, and joins
description: One-page reference for ObjectStack QuerySchema — filters, sorts, pagination, aggregations, and expand
---

# Query Syntax Cheat Sheet
Expand Down Expand Up @@ -387,47 +387,27 @@ on it either. Post-filter grouped results on the client until this is wired up.

---

## Joins
## Joins — removed

<Callout type="warn">
`joins` is defined in `QuerySchema` (`JoinNodeSchema`) and the SQL driver advertises
`supports.joins: true`, but no driver's `find()` actually executes a join today — the SQL,
in-memory, and MongoDB drivers all ignore the `joins` array (there is no `.join()`/`.leftJoin()`
call in the SQL driver's query builder). Use `expand` for relationship traversal instead.
`query.joins` was **removed in `@objectstack/spec` 18** (#4286, ADR-0049
enforce-or-remove): no driver's `find()` ever executed a join — the SQL, in-memory, and
MongoDB drivers all ignored the array, so it only ever declared a capability that did
not run. The key is tombstoned: authoring it is a `tsc` error, and a query carrying it
(even `joins: []`) fails to parse with the upgrade prescription. The
`JoinNode`/`JoinType`/`JoinStrategy` exports left with it.
</Callout>

### Join Types

| Type | Description |
|:---|:---|
| `inner` | Records with matches in both objects |
| `left` | All records from left + matching from right |
| `right` | All records from right + matching from left |
| `full` | All records from both objects |

### Join Strategies

| Strategy | Description |
|:---|:---|
| `auto` | Let the engine choose the best strategy |
| `database` | Push join to the database level |
| `hash` | In-memory hash join |
| `loop` | Nested loop join |

### Join Example
Use `expand` for relationship traversal — the live spelling for related records — or a
dotted `fields` path (`'customer.name'`) for a single related column:

```typescript
{
object: 'order',
fields: ['id', 'total', 'customer.name', 'customer.email'],
joins: [
{
type: 'inner',
object: 'customer',
on: ['order.customer_id', '=', 'customer.id'],
strategy: 'auto'
}
]
fields: ['id', 'total'],
expand: {
customer_id: { object: 'customer', fields: ['name', 'email'] }
}
}
```

Expand All @@ -440,13 +420,7 @@ call in the SQL driver's query builder). Use `expand` for relationship traversal
object: 'article',
search: {
query: 'kubernetes deployment',
fields: ['title', 'body', 'tags'],
fuzzy: true,
operator: 'and',
boost: { title: 2.0, body: 1.0 },
minScore: 0.5,
language: 'english',
highlight: true
fields: ['title', 'body', 'tags']
}
}
```
Expand All @@ -455,19 +429,20 @@ call in the SQL driver's query builder). Use `expand` for relationship traversal
|:---|:---|:---|
| `query` | `string` | Search text |
| `fields` | `string[]` | Fields to search (optional — defaults to all searchable) |
| `fuzzy` | `boolean` | Enable fuzzy matching for typo tolerance |
| `operator` | `'and' \| 'or'` | How to combine search terms |
| `boost` | `Record<string, number>` | Field relevance weights |
| `minScore` | `number` | Minimum relevance score (0–1) |
| `language` | `string` | Language for stemming/stopwords |
| `highlight` | `boolean` | Return highlighted matches |
| `fuzzy` | `boolean` | `[EXPERIMENTAL — not enforced]` Fuzzy matching for typo tolerance |
| `operator` | `'and' \| 'or'` | `[EXPERIMENTAL — not enforced]` How to combine search terms |
| `boost` | `Record<string, number>` | `[EXPERIMENTAL — not enforced]` Field relevance weights |
| `minScore` | `number` | `[EXPERIMENTAL — not enforced]` Minimum relevance score (0–1) |
| `language` | `string` | `[EXPERIMENTAL — not enforced]` Language for stemming/stopwords |
| `highlight` | `boolean` | `[EXPERIMENTAL — not enforced]` Return highlighted matches |

<Callout type="warn">
Only `query` and `fields` are implemented. The engine expands `search` into a driver-agnostic
`$and`-of-`$or`-of-`$contains` filter (ADR-0061) — `fuzzy`, `operator`, `boost`, `minScore`,
`language`, and `highlight` are accepted by `QuerySchema` but read nowhere in
`expandSearchToFilter()` / `normalizeSearch()`, so they have no effect. Multiple search terms
are always AND-ed regardless of `operator`.
`expandSearchToFilter()` / `normalizeSearch()`, so they have no effect; since #4286 their
`.describe()` markers say so. Multiple search terms are always AND-ed regardless of
`operator`.
</Callout>

### Pinyin recall (Chinese deployments)
Expand Down Expand Up @@ -496,46 +471,36 @@ by `@objectstack/plugin-pinyin-search`) recomputes the column on demand.

---

## Window Functions
## Window Functions — removed from the request surface

<Callout type="warn">
`windowFunctions` is only reachable by calling the SQL driver's `findWithWindowFunctions()`
method directly. `ObjectQL.find()` / `.aggregate()` and the `POST /api/v1/data/:object/query`
route never call it, so sending `windowFunctions` through the standard client/REST query path
has no effect.
`query.windowFunctions` was **removed in `@objectstack/spec` 18** (#4286):
`ObjectQL.find()` / `.aggregate()` and the `POST /api/v1/data/:object/query` route never
routed it anywhere, so sending it had no effect. The key is tombstoned — a query carrying
it fails to parse with the upgrade prescription — and the
`WindowFunction`/`WindowSpec`/`WindowFunctionNode` exports left with it.
</Callout>

| Function | Description |
|:---|:---|
| `row_number` | Sequential row number within partition |
| `rank` | Rank with gaps for ties |
| `dense_rank` | Rank without gaps for ties |
| `percent_rank` | Relative rank as percentage |
| `lag` | Access previous row value |
| `lead` | Access next row value |
| `first_value` | First value in window |
| `last_value` | Last value in window |
| `sum` / `avg` / `count` / `min` / `max` | Running aggregations |

### Window Function Example
Window functions remain a **SQL-driver door**: `findWithWindowFunctions()`, callable
directly on a SQL driver instance (it is not on the `IDataDriver` contract). Its input
is the driver's own flat shape:

```typescript
{
object: 'employee',
fields: ['name', 'department', 'salary'],
const ranked = await sqlDriver.findWithWindowFunctions('employee', {
windowFunctions: [
{
function: 'rank',
alias: 'salary_rank',
over: {
partitionBy: ['department'],
orderBy: [{ field: 'salary', order: 'desc' }]
}
partitionBy: ['department'],
orderBy: [{ field: 'salary', order: 'desc' }]
}
]
}
});
```

For request-level analytics, use `aggregations` + `groupBy`, or model rankings in
report/dashboard metadata.

---

## Distinct & Group By
Expand Down
2 changes: 1 addition & 1 deletion content/docs/protocol/objectql/index.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -396,6 +396,6 @@ See [Security Protocol](/docs/protocol/objectql/security) for details.
<Card
title="Query Syntax"
href="/docs/protocol/objectql/query-syntax"
description="Write database-agnostic queries with filters, joins, and aggregations"
description="Write database-agnostic queries with filters, expand, and aggregations"
/>
</Cards>
78 changes: 39 additions & 39 deletions content/docs/protocol/objectql/query-syntax.mdx
Original file line number Diff line number Diff line change
Expand Up @@ -70,11 +70,9 @@ interface QueryAST {
offset?: number; // Skip records (OFFSET)
top?: number; // Alias for limit (OData compat)
cursor?: Record<string, unknown>; // Keyset pagination cursor
joins?: JoinNode[]; // Explicit JOINs
aggregations?: AggregationNode[]; // Aggregation functions
groupBy?: GroupByNode[]; // GROUP BY targets (string | object)
having?: FilterCondition; // HAVING clause
windowFunctions?: WindowFunctionNode[]; // Window functions (OVER)
distinct?: boolean; // SELECT DISTINCT
expand?: Record<string, QueryAST>; // Recursive relation loading
}
Expand All @@ -92,14 +90,22 @@ on the `find()` path:

| Member | Status |
|:-------|:-------|
| `joins` | No driver reads it — there is no `query.joins` / `ast.joins` consumer anywhere in `packages/` |
| `having` | Never read: `engine.aggregate()` forwards only `object`/`where`/`groupBy`/`aggregations`, and the in-memory fallback has no HAVING stage |
| `cursor` | Accepted by `EngineQueryOptions`, but no driver implements keyset pagination |
| `distinct` | Not applied by `find()`; the SQL and in-memory drivers expose a separate `distinct(object, field, filters?)` method instead |
| `windowFunctions` | Not applied by `find()`; `SqlDriver.findWithWindowFunctions()` is a separate method that is not on the `IDataDriver` contract |
| `search.fuzzy` / `boost` / `operator` / `minScore` / `language` / `highlight` | Parsed by the schema but ignored: only `query` and `fields` drive the expansion |
| `aggregations[].filter` | `[EXPERIMENTAL — not enforced]` — a SQL `FILTER (WHERE …)` affordance neither the SQL builders nor the in-memory fallback applies |
| `search.fuzzy` / `boost` / `operator` / `minScore` / `language` / `highlight` | `[EXPERIMENTAL — not enforced]` — only `query` and `fields` drive the expansion |

`top` is the exception that *is* honored — the engine normalises it to `limit`.

Two members left this table by being **removed** (#4286, ADR-0049 enforce-or-remove):
`joins` and `windowFunctions` are tombstoned in `@objectstack/spec` 18 — a query
carrying either fails to parse with the upgrade prescription, and authoring one is a
`tsc` error. Related records are read through `expand`; window functions remain a
SQL-driver door (`SqlDriver.findWithWindowFunctions()`). The inert members that
*remain* declared (`having`, `cursor`, `distinct`, and the experimental flags above)
are tracked in the liveness ledger (`packages/spec/liveness/query.json`) pending
their #4286 dispositions.
</Callout>

### Key Types
Expand All @@ -118,7 +124,7 @@ interface AggregationNode {
field?: string; // optional for COUNT(*)
alias: string; // result column alias
distinct?: boolean; // DISTINCT before aggregation — in-memory path only
filter?: FilterCondition; // FILTER WHERE clause — protocol only, never applied
filter?: FilterCondition; // [EXPERIMENTAL — not enforced] FILTER WHERE clause — never applied
}

// FieldNode — one entry of the select list. A field name, optionally dotted to
Expand Down Expand Up @@ -742,51 +748,43 @@ expansion emits a plain `$contains`, which `SqlDriver` compiles to a parameteris
`LIKE '%…%'` with no case folding — so the dialect's own `LIKE`/collation rules decide —
while the in-memory driver matches with a case-insensitive regex. Only `select` /
`status` option *labels* are matched case-insensitively by the expansion itself.
`fuzzy`, `boost`, `operator`, `minScore`, `language`, and `highlight` are accepted by
the schema but ignored by the expansion.

### Joins (protocol only)
`fuzzy`, `boost`, `operator`, `minScore`, `language`, and `highlight` carry
`[EXPERIMENTAL — not enforced]` markers (#4286): the schema accepts them, the
expansion ignores them.

`QuerySchema` defines a `joins` array, but **no driver reads it** — there is no
`query.joins` consumer anywhere in `packages/`, so a query carrying `joins` silently
runs as a single-table query. Use `expand` (§4) for relationship loading, or two
queries joined in application code.
### Joins — removed (#4286)

For reference, the protocol shape is:
`query.joins` was **removed in `@objectstack/spec` 18** (#4286, ADR-0049
enforce-or-remove): no driver ever read it, so a query carrying `joins` silently ran
as a single-table query. The key is tombstoned — authoring it is a `tsc` error, and a
query that still carries it (even as an empty array) fails to parse with the upgrade
prescription. The `JoinNode` / `JoinType` / `JoinStrategy` exports left with it.

```typescript
const query: QueryAST = {
object: 'order',
fields: ['id', 'amount'],
joins: [
{
type: 'inner',
object: 'customer',
alias: 'c',
on: { 'order.customer_id': { $eq: { $field: 'c.id' } } },
},
],
};
```
Use `expand` (§4) for relationship loading — the live spelling for related records —
a dotted `fields` path (`'owner.name'`) for a single related column, or two queries
joined in application code.

**Join types:** `inner`, `left`, `right`, `full`
### Window Functions — removed from the request surface (#4286)

### Window Functions
`query.windowFunctions` was **removed in `@objectstack/spec` 18** (#4286): `find()`
never applied it, so every OVER clause it declared was silently dropped. The key is
tombstoned, and the `WindowFunction` / `WindowSpec` / `WindowFunctionNode` exports
left with it — they declared `field` / `over` / `frame` members that no executor ever
read.

`windowFunctions` on a `QueryAST` is ignored by `find()`. The SQL drivers implement
window functions through a separate `findWithWindowFunctions()` method, which is not on
the `IDataDriver` contract and is not surfaced by `IDataEngine`:
Window functions remain a **SQL-driver door**: `findWithWindowFunctions()`, which is
not on the `IDataDriver` contract and is not surfaced by `IDataEngine`. Its input is
the driver's own flat shape — function name, alias, and optional flat
`partitionBy` / `orderBy`:

```typescript
const ranked = await driver.findWithWindowFunctions('order', {
windowFunctions: [
{
function: 'row_number',
alias: 'rank',
over: {
partitionBy: ['customer_id'],
orderBy: [{ field: 'amount', order: 'desc' }],
},
partitionBy: ['customer_id'],
orderBy: [{ field: 'amount', order: 'desc' }],
},
],
});
Expand All @@ -796,7 +794,9 @@ const ranked = await driver.findWithWindowFunctions('order', {
```

That method always selects `*` plus the window columns — it does not honor a `fields`
projection.
projection, and niladic rendering means argument-taking functions (`LAG(field)`)
emit without their argument. For request-level analytics use `aggregations` +
`groupBy` (§5).

---

Expand Down
Loading
Loading