fix(spec,plugins): a paged read with no orderBy is a partition too — the shape every list view actually sends (#4363)

.changeset/paged-read-determinism.md

@@ -39,7 +39,7 @@ The obligation is now normative on `IDataDriver.find`, with shared cases in
 `@objectstack/spec/data` (`PAGINATION_CASES`) that all three drivers run — so a
 future driver is held to it by a gate rather than by remembering.
 
-Deliberately not covered: a paged read with **no** `orderBy`. That is
-non-deterministic on every backend by definition and imposing an order on
-callers who asked for none changes plan selection far more broadly; filed as
-#4363.
+Not covered by this change: a paged read with **no** `orderBy`. Same defect,
+wider blast radius, so it was carved out to #4363 rather than folded in — and
+closed there, in the same release. The contract, the shared cases and both
+drivers now cover a paged read whatever its `orderBy`, including none at all.

.changeset/unordered-paged-read-determinism.md

@@ -0,0 +1,63 @@
+---
+"@objectstack/spec": patch
+"@objectstack/driver-sql": patch
+"@objectstack/driver-mongodb": patch
+---
+
+fix(data): a paged read with no `orderBy` is a partition too — the shape every list view actually sends (#4363)
+
+objectui#3106's server half closed the **sorted** paged read: a non-empty
+`orderBy` now carries a unique tie-breaker, so `ORDER BY status LIMIT 50 OFFSET
+50` can no longer serve one row twice while never serving another. It stopped
+there deliberately. This closes the half it left, which is the more common one.
+
+A list view whose metadata configures no `sort`, on which nobody has clicked a
+column header, sends no `$orderby` at all. `SqlDriver` and `MongoDBDriver` then
+emitted a bare `LIMIT`/`OFFSET` — and neither backend promises anything about
+the order that slices:
+
+- **SQL** leaves the row order of an unordered read to the plan. Small tables
+  hand back insertion order in practice, which is exactly why this survives
+  testing; a parallel scan, an index scan, or a `VACUUM` need not.
+- **MongoDB** returns natural order, which describes where a document currently
+  sits in its extent — and moves when the document does.
+
+Every row ties with every other on an empty sort key, so this is the same defect
+at full strength rather than a different one: page 2 repeats a row page 1 showed
+and drops one nobody sees, with every page full and every row real.
+
+Both drivers now order a paged read by their unique key column when the caller
+supplied no sort keys — the same `id` the tie-breaker was already appending, now
+standing alone. `driver-memory` again needed no change: it slices its backing
+array, and two reads with no write between them see the identical sequence. The
+contract asks for a partition, not for id order.
+
+**Unpaged reads are untouched, deliberately.** The rule keys off `limit`/
+`offset`, not off `orderBy` being absent. A read with neither hands back the
+whole matching set, so no caller can be shown a partial view of it, and sorting
+every read in the system would change plan selection to buy nothing. `limit`
+alone does count as paged: page one of a walk is routinely `limit=50` with no
+offset, and ordering only the later pages would leave the defect fully intact.
+
+`SqlDriver` keeps the existing restriction to objects it created itself
+(`initObjects` records them). It matters more here than for the sorted case: on
+a federated table (ADR-0015) there is no requested sort for #3821's ladder to
+fall back to, so a wrong guess about `id` would turn a reshuffle into a failed
+read. Those tables now get a warning — once per object, behavior unchanged —
+because the contract states determinism as a MUST, and a MUST that quietly does
+not hold is the same invisible failure the rule was written against.
+
+`findOne` is deliberately outside all of this, and the contract now says so.
+Engines reach a driver with `limit: 1`, which is shaped exactly like page one of
+a walk, but it promises *a* matching record rather than a position in a
+sequence — nothing for a second call to be inconsistent with. Reading it as a
+page would put `ORDER BY id LIMIT 1` on the hottest read in the system, which is
+the classic shape for a planner to abandon the predicate's own index: measured
+on Postgres 16 over 2M rows, `WHERE owner_id = ? LIMIT 1` went 0.08 ms → 7.8 ms
+and swapped the `owner_id` index for the primary key. `MongoDBDriver.findOne`
+has never sorted, so this also puts the two drivers back in step.
+
+The obligation is normative on `IDataDriver.find` and the cases are shared —
+`PAGINATION_UNORDERED_CASES` alongside `PAGINATION_CASES` in
+`@objectstack/spec/data` — so a future driver is held to both halves by a gate
+rather than by remembering.

content/docs/data-modeling/queries.mdx

@@ -239,6 +239,21 @@ reaching `engine.find()` directly are unaffected.
 }
 ```
 
+<Callout type="info">
+**Walking the pages visits every row exactly once**, whatever you sort by — or
+whether you sort at all. The SQL and MongoDB drivers get there by ordering on a
+unique column of their own, on top of whatever `orderBy` you gave, because a
+sort key like `status` does not identify a row and no backend promises equal
+keys keep the same arrangement between two queries. Without that, page 2 repeats
+a row page 1 already showed and skips one nobody ever sees — every page full,
+every row real, the two halves of the symptom several screens apart
+(objectui#3106, #4363).
+
+The guarantee attaches to `limit`/`offset`, so it costs nothing on the reads
+that do not paginate: a query with neither is returned in whatever order the
+backend chooses, exactly as before.
+</Callout>
+
 ### Keyset Pagination — a `where` predicate on the sort key
 
 `query.cursor` was **removed in `@objectstack/spec` 18** (#4286): nothing on the server

content/docs/protocol/objectql/query-syntax.mdx

@@ -859,6 +859,15 @@ const page2 = await engine.find('customer', {
 
 **Drawback:** Slow for large offsets (database still scans all skipped rows).
 
+**Determinism:** the walk above visits every `customer` exactly once even though
+it names no `orderBy`. A driver orders any paged read by a unique column of its
+own — appended to your sort keys, or standing alone when you gave none — because
+an unordered `LIMIT`/`OFFSET` slices an arrangement nothing holds steady: SQL
+leaves row order to the plan, and MongoDB's natural order moves when a document
+does. Page 2 would otherwise repeat a row from page 1 and drop another, with
+every page full and every row real (objectui#3106, #4363). A query with no
+`limit`/`offset` is untouched — nothing is being sliced, so no order is imposed.
+
 ### Keyset Pagination
 
 <Callout type="warn">

packages/plugins/driver-memory/src/memory-pagination-conformance.test.ts

@@ -1,27 +1,41 @@
 // Copyright (c) 2026 ObjectStack. Licensed under the Apache-2.0 license.
 
 /**
- * Deterministic paged reads for the in-memory driver (objectui#3106) — the
- * contract on `IDataDriver.find`, checked against the shared cases in
- * `@objectstack/spec/data`.
+ * Deterministic paged reads for the in-memory driver (objectui#3106,
+ * objectstack#4363) — the contract on `IDataDriver.find`, checked against the
+ * shared cases in `@objectstack/spec/data`.
  *
- * This driver needed **no change** to satisfy it, and that is worth a suite
- * rather than a shrug. It sorts with `Array#sort`, which ES2019 onward
- * guarantees is stable, over a table array whose order does not move between
- * two reads — so equal keys keep the same relative arrangement on page 2 that
- * they had on page 1, which is precisely what the contract asks for.
+ * This driver needed **no change** to satisfy either half, and that is worth a
+ * suite rather than a shrug.
+ *
+ * - **Sorted pages.** It sorts with `Array#sort`, which ES2019 onward
+ *   guarantees is stable, over a table array whose order does not move between
+ *   two reads — so equal keys keep the same relative arrangement on page 2 that
+ *   they had on page 1.
+ * - **Unsorted pages.** With no `orderBy` it slices that same array directly,
+ *   and the array is the storage: two reads with no write between them see the
+ *   identical sequence. There is no plan to change its mind and no natural
+ *   order to move, so the walk partitions the set without a sort being imposed.
+ *   The contract asks for determinism, not for id order, and this driver
+ *   supplies it from a different direction than SQL and MongoDB do.
  *
  * The guarantee is therefore load-bearing but implicit: it rests on `sort`
- * being stable and on `applySort` copying rather than reordering the table in
- * place. Both are easy to lose in a refactor that looks like a speed-up — a
- * hand-rolled quicksort, or sorting the backing array directly — and neither
- * loss would fail any other test in this package. That is what this file is
- * for: it holds the property against the day the implementation changes, which
- * is the only day it could break.
+ * being stable, on `applySort` copying rather than reordering the table in
+ * place, and on `find` slicing a copy rather than the live table. All are easy
+ * to lose in a refactor that looks like a speed-up — a hand-rolled quicksort,
+ * sorting the backing array directly, a "reuse the array" allocation saving —
+ * and no other test in this package would fail. That is what this file is for:
+ * it holds the property against the day the implementation changes, which is
+ * the only day it could break.
  */
 
 import { describe, it, expect, beforeEach } from 'vitest';
-import { PAGINATION_ALL_IDS, PAGINATION_CASES, PAGINATION_ROWS } from '@objectstack/spec/data';
+import {
+  PAGINATION_ALL_IDS,
+  PAGINATION_CASES,
+  PAGINATION_ROWS,
+  PAGINATION_UNORDERED_CASES,
+} from '@objectstack/spec/data';
 import { InMemoryDriver } from './memory-driver.js';
 
 describe('InMemoryDriver — paged reads are a partition of the result set (objectui#3106)', () => {
@@ -67,4 +81,33 @@ describe('InMemoryDriver — paged reads are a partition of the result set (obje
       expect(paged.map((r) => r.id)).toEqual(whole.map((r: any) => r.id));
     });
   }
+
+  for (const testCase of PAGINATION_UNORDERED_CASES) {
+    it(`visits every row exactly once with NO orderBy at all — ${testCase.name}`, async () => {
+      const seen: string[] = [];
+      for (let offset = 0; offset < PAGINATION_ROWS.length; offset += testCase.pageSize) {
+        const page = await driver.find('ticket', { limit: testCase.pageSize, offset } as any);
+        seen.push(...page.map((r: any) => String(r.id)));
+      }
+
+      expect(seen).toHaveLength(PAGINATION_ALL_IDS.length);
+      expect(new Set(seen).size).toBe(PAGINATION_ALL_IDS.length);
+      expect([...seen].sort()).toEqual([...PAGINATION_ALL_IDS].sort());
+    });
+
+    it(`page boundaries are invisible with NO orderBy — ${testCase.name}`, async () => {
+      const paged: any[] = [];
+      for (let offset = 0; offset < PAGINATION_ROWS.length; offset += testCase.pageSize) {
+        const page = await driver.find('ticket', { limit: testCase.pageSize, offset } as any);
+        paged.push(...page);
+      }
+
+      // Concatenating the pages reproduces the unpaged read exactly. Note this
+      // is insertion order, NOT id order: the contract asks for a partition,
+      // and this driver's storage already provides one, so nothing is imposed.
+      const whole = await driver.find('ticket', {} as any);
+      expect(paged.map((r) => r.id)).toEqual(whole.map((r: any) => r.id));
+      expect(paged.map((r) => r.id)).toEqual(PAGINATION_ROWS.map((r) => r.id));
+    });
+  }
 });

packages/plugins/driver-mongodb/src/mongodb-driver.ts

@@ -238,7 +238,7 @@ export class MongoDBDriver implements IDataDriver {
     }
 
     // Sorting
-    const sort = this.buildSortSpec(query.orderBy);
+    const sort = this.buildSortSpec(query);
     if (sort) findOptions.sort = sort;
 
     // Pagination
@@ -277,7 +277,7 @@ export class MongoDBDriver implements IDataDriver {
       projection: { _id: 0 },
     };
 
-    const sort = this.buildSortSpec(query.orderBy);
+    const sort = this.buildSortSpec(query);
     if (sort) findOptions.sort = sort;
 
     if (query.offset !== undefined) findOptions.skip = query.offset;
@@ -609,27 +609,46 @@ export class MongoDBDriver implements IDataDriver {
    * silently drops another — with every page full, every row real, and the two
    * halves of the symptom too far apart for anyone to notice.
    *
+   * A paged read with **no** `sort` is the same defect at full strength
+   * (objectstack#4363), which is why this reads the whole query rather than
+   * just its `orderBy`. Unsorted documents come back in natural order, and
+   * natural order describes where a document currently sits in its extent — it
+   * moves when the document does, so page 2 of a walk can be cut from a layout
+   * page 1 no longer describes. The empty sort key is simply the case where
+   * every document ties with every other, so the same `id` suffix that was
+   * separating one `status` group ends up carrying the entire order.
+   *
    * `id` is always present (`create()` fills it when the caller omits one), so
-   * unlike the SQL driver there is no table this cannot apply to. It is
-   * appended in the LAST requested key's direction: determinism holds either
-   * way, but a same-direction suffix is the one a compound index can still walk
-   * in a single pass.
+   * unlike the SQL driver there is no collection this cannot apply to, and
+   * `syncCollectionSchema` gives every collection it provisions a unique
+   * `idx_id_unique` — so a sort that ends in `id` is index-served rather than
+   * a blocking in-memory sort against the 100 MB cap. It is appended in the
+   * LAST requested key's direction (`1` when there is none): determinism holds
+   * either way, but a same-direction suffix is the one a compound index can
+   * still walk in a single pass.
    *
-   * Returns `undefined` when nothing was requested — an unordered read stays
-   * unordered (see the contract's explicit carve-out).
+   * Returns `undefined` for a read that is neither sorted nor paged — nothing
+   * is being sliced there, so a caller who asked for no order keeps none (the
+   * contract's explicit carve-out).
    */
-  private buildSortSpec(orderBy: QueryAST['orderBy']): Document | undefined {
-    if (!orderBy || !Array.isArray(orderBy)) return undefined;
+  private buildSortSpec(query: QueryAST): Document | undefined {
     const sort: Document = {};
     let lastDirection: 1 | -1 = 1;
-    for (const item of orderBy) {
-      if (item.field) {
-        lastDirection = item.order === 'desc' ? -1 : 1;
-        sort[this.mapFieldName(item.field)] = lastDirection;
+    if (Array.isArray(query.orderBy)) {
+      for (const item of query.orderBy) {
+        if (item.field) {
+          lastDirection = item.order === 'desc' ? -1 : 1;
+          sort[this.mapFieldName(item.field)] = lastDirection;
+        }
       }
     }
-    if (Object.keys(sort).length === 0) return undefined;
-    if (sort.id === undefined) sort.id = lastDirection;
+
+    const requested = Object.keys(sort).length > 0;
+    const paged = query.limit !== undefined || query.offset !== undefined;
+    if (!requested && !paged) return undefined;
+
+    const idKey = this.mapFieldName('id');
+    if (sort[idKey] === undefined) sort[idKey] = lastDirection;
     return sort;
   }