fix(spec,plugins): a paged read with no orderBy is a partition too — the shape every list view actually sends (#4363) - #4378
Merged
Conversation
… five queries that share a WHERE clause (objectui#3106) `ORDER BY status LIMIT 50 OFFSET 50` names a sort key that does not identify a row, and no backend promises that rows with equal keys keep the same relative arrangement between two queries. MongoDB documents this outright: `sort` + `skip`/`limit` on a non-unique key may return the same document more than once. Page 2 then repeats a row page 1 already showed and skips one nobody ever sees — with every page full, every row real, and the two halves of the symptom several screens apart. SqlDriver and MongoDBDriver append a unique tie-breaker to any non-empty `orderBy`, in the last requested key's direction: determinism holds either way, but a same-direction suffix is the one an index can still walk in a single pass. SqlDriver applies it only to objects it created itself (`initObjects` records those in `managedObjectFields`). A federated table (ADR-0015) may carry no `id` column, and guessing there would be worse than doing nothing — the resulting unknown-column error is answered by #3821's recovery ladder retrying with NO ORDER BY at all, trading a reshuffle among ties for the loss of the caller's whole sort. driver-memory needed no change: `Array#sort` is stable and the backing table's order does not move between reads. It gets a suite anyway, because that guarantee is implicit and is exactly what a refactor that looks like a speed-up (a hand-rolled sort, or sorting the array in place) would silently remove. The obligation is normative on `IDataDriver.find` and the cases are shared (`PAGINATION_CASES` in `@objectstack/spec/data`), so a future driver is held to it by a gate rather than by remembering. A paged read with NO `orderBy` is deliberately out of scope and filed as #4363. Co-Authored-By: Claude Fable 5 <noreply@anthropic.com>
…-breaker-missing-p98qf0
…4363) objectui#3106's server half closed the SORTED paged read: a non-empty `orderBy` carries a unique tie-breaker, so `ORDER BY status LIMIT 50 OFFSET 50` can no longer serve one row twice while never serving another. It stopped there deliberately. This closes the half it left, which is the more common one. A list view whose metadata configures no `sort`, on which nobody has clicked a column header, sends no `$orderby` at all. `SqlDriver` and `MongoDBDriver` then emitted a bare LIMIT/OFFSET, and neither backend promises anything about the order that slices: SQL leaves the row order of an unordered read to the plan, and MongoDB's natural order moves when a document does. Every row ties with every other on an empty sort key, so it is the same defect at full strength rather than a different one. Both drivers now order a paged read by their unique key column when the caller supplied no sort keys -- the same `id` the tie-breaker was already appending, standing alone. `driver-memory` again needed no change: it slices its backing array, and two reads with no write between them see the identical sequence. The contract asks for a partition, not id order. Unpaged reads are untouched, deliberately: the rule keys off `limit`/`offset`, not off `orderBy` being absent. `limit` alone does count -- page one of a walk is routinely `limit=50` with no offset, and ordering only the later pages would leave the defect fully intact. `findOne` is outside all of this, and the contract now says so. Engines reach a driver with `limit: 1`, shaped exactly like page one of a walk, but it promises A matching record rather than a position in a sequence. Reading it as a page would put `ORDER BY id LIMIT 1` on the hottest read in the system, the classic shape for a planner to abandon the predicate's own index: measured on Postgres 16 over 2M rows, `WHERE owner_id = ? LIMIT 1` went 0.08ms -> 7.8ms and swapped the owner_id index for the primary key. `MongoDBDriver.findOne` has never sorted, so this also puts the two drivers back in step. Federated tables (ADR-0015) keep the existing carve-out -- no ordering column can be guessed there -- but now say so once per object, because the contract states determinism as a MUST and a MUST that quietly does not hold is the failure the rule was written against. The obligation is normative on `IDataDriver.find` and the cases are shared (`PAGINATION_UNORDERED_CASES` beside `PAGINATION_CASES`), so a future driver is held to both halves by a gate rather than by memory. Also repairs two holes in the tie-breaker suites this builds on: `InspectableSqlDriver` reimplemented the ORDER BY instead of observing the one `find()` emits (green on the day `find()` stops calling it), and MongoDB's pure sort-spec assertions sat behind a `skipIf` on a server they do not need -- so the only check that catches a deleted feature went silent, and reported green, wherever the mongod binary cannot be fetched. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016niiseyRn43L9M4JdAwuqD
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
…-breaker-missing-p98qf0 # Conflicts: # .changeset/paged-read-determinism.md # packages/plugins/driver-memory/src/memory-pagination-conformance.test.ts # packages/plugins/driver-mongodb/src/mongodb-driver.ts # packages/plugins/driver-mongodb/src/mongodb-pagination-conformance.test.ts # packages/plugins/driver-sql/src/sql-driver-pagination-conformance.test.ts # packages/plugins/driver-sql/src/sql-driver.ts # packages/spec/api-surface.json # packages/spec/src/contracts/data-driver.ts # packages/spec/src/data/index.ts # packages/spec/src/data/pagination-conformance.ts
github-actions
Bot
added
size/l
documentation
Contributor
📓 Docs Drift CheckThis PR changes 3 package(s): 109 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
|
…-breaker-missing-p98qf0
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Closes #4363。#4367 收口了「有
orderBy的分页」，本 PR 收口它按 Prime Directive #10 拆出去的另一半。决策：走方向 1，并且量过了
issue 列了三个方向、倾向 1（「
orderBy为空且limit/offset存在时按主键排序」），但要求「在真实数据量上先量一次计划变化再定」。量了 —— Postgres 16、2,000,000 行、390 MB、VACUUM FULL ANALYZE之后、热缓存、best of 3：limit 50ticket_pkeylimit 50 offset 10000ticket_pkeylimit 50 offset 1000000ticket_pkeylimit 50ticket_pkeylimit 50ticket_owner_idx不变，加 top-N sortlimit 5ticket_owner_idx不变，加 top-N sortfindOne形态limit 1ticket_owner_idx→ticket_pkey（放弃谓词自己的索引）结论分两条：
分页读取本身可以接受。 谓词有索引时计划不变，只多一个 top-N sort（毫秒级）；无谓词时从顺序扫描换成主键索引扫描，代价随 offset 线性上升。深翻页那一格（1.09 s）确实难看，但它 before 就已经是 70 ms 的病态形态，文档本来就建议改用 keyset —— 而且它 before 之所以快，正是因为它返回的是错的结果。这是把「快而不确定」换成「慢而确定」，方向没得选。
limit 1是另一回事，所以我把findOne排除在外了。 这一条 issue 没预见到：engine.findOne()按契约写死limit: 1，而SqlDriver.findOne()是转调find()实现的 —— 于是「分页读取」这条规则会顺手盖住全系统最热的那条读路径。ORDER BY id LIMIT 1恰好是让 planner 判断「反正很快就能停」从而放弃谓词索引、改走主键加过滤的经典形态：上表最后一行，约 100×。而且这里什么都换不到：
findOne承诺的是「一条匹配的记录」，不是「序列中的某个位置」，没有第二页可以跟它不一致。所以find与findOne从共享的findRows()分叉，singleRowLookup只影响这一件事。顺带的好处是这让两个 driver 重新对齐 ——MongoDBDriver.findOne走collection.findOne，从来就没排过序。契约里也写下了这条边界（
IDataDriver.findOne的 JSDoc），免得下一个人把它当成漏掉的。没走 2 和 3 的理由。 方向 2（全量补默认排序）上表已经回答了：谓词有索引的读取要多付一个 top-N sort，无谓词的要从顺序扫描换成主键索引扫描 —— 而不分页的读取把整个匹配集交给调用方，没有任何一页可以是错的，这笔钱买不到东西。方向 3（在 protocol 边界对「
top/skip存在而sort缺失」告警或拒绝）把决定权交回调用方，姿态上和 #4226 同源，但它要求每一个现存调用方都改：而这些调用方今天并没有做错事 —— 「不关心顺序」是一个合法的意图，只是它不该顺带意味着「可以少给我一行」。真正错的是我们把「不关心顺序」实现成了「不保证是个划分」。所以修在 driver，理由和 tie-breaker 一样：哪一列唯一、表上到底有没有，只有 driver 知道。改了什么
契约（
IDataDriver.find） —— 义务从「orderBy非空 + 分页」放宽到「只要分页」，两种破坏形态并列写出：不能标识行的排序键，和根本没有排序键（后者是前者的退化情形 —— 空排序键上每一行都跟每一行并列）。原来那段「Deliberately NOT covered」换成真正剩下的那个 carve-out：不分页的无序读取。SqlDriver—— ORDER BY 的构造收进orderKeysFor()，三种形态一张表：非空orderBy→ 调用方的键 +id；空orderBy+ 分页 → 只有id；空orderBy+ 不分页 → 什么都不加。limit单独出现也算分页：列表第一页常常就是limit=50不带 offset，只给带 offset 的页排序等于让第 1 页跟后面的页切自不同的排列 —— bug 原封不动，外面套一层像修好了的样子。#3821 的恢复梯子也跟着改问「这条语句到底有没有 ORDER BY」，而不再问「调用方给没给」—— 自本 PR 起这是两个问题。MongoDBDriver——buildSortSpec()改读整个 query 而非只读orderBy。无sort时文档按 natural order 返回，而 natural order 描述的是文档当前在 extent 里的位置，文档一动它就变。syncCollectionSchema给每个它建的集合都建了唯一索引idx_id_unique，所以以id收尾的 sort 走索引，不会撞 100 MB 的内存排序上限。driver-memory又一次不用改 —— 它切的是自己的存储数组，两次读之间没有写就是同一个序列。契约要的是「划分」，不是「按 id 排」，这个 driver 从另一个方向满足它。给它补了用例，理由跟 #4367 给它补用例一样：这个保证是隐式的，一次看起来像提速的重构就会丢掉，而且不会让这个包里任何别的测试变红。联邦表现在会出声。
SqlDriver保留 #4367 的限制（只对自己建的表加序），但这里的代价比 sorted 那半大：sorted 的读取至少还留着调用方自己的 ORDER BY，只是其中的并列会重排；unsorted 的读取则一无所有。行为不变，但每个对象报一次 warn —— 契约现在把确定性写成 MUST，一个静悄悄不成立的 MUST 就是这条规则本来要消灭的那种失败。测试
合并
main（含刚落地的 #4367 与 #4373）之后pnpm install --frozen-lockfile+ 全量pnpm build+ 清.objectstack/，再全部重跑：pnpm testdogfood74 files、cli、example-showcase）—— 合并前跑过一次，合并后再跑一次driver-sql全量driver-memory全量driver-sqlite-wasm全量SqlDriver，同一条代码路径）driver-mongodb全量@objectstack/spec全量objectql/rest/runtime@objectstack/spec check:generatedapi-surface.json重新生成，恰好 +2 行 = 两个新导出）check:liveness/check:exported-any/check:type-check-coverage/check:doc-authoring/check:role-word/check:nul-bytestypecheck（spec、driver-mongodb）+ 改动文件 eslintdriver-sql/driver-memory在覆盖率台账里无typecheckscript）两处 mutation check（不是「跑绿了」而是「拆了会红」）：把
SqlDriver里的paged钉成false→ 5 个用例红；把MongoDBDriver的同一处钉成false→ 1 个红。顺手修了 #4367 两个测试上的洞，因为我正好要往它们上面加断言：
InspectableSqlDriver.orderByClauseFor()是在测试里重新实现了一遍 tie-breaker 逻辑再编译 SQL —— 也就是说find()哪天不再调用它，这个断言照样绿，而那个文件的头注释写的恰恰是「clause 断言是删掉功能时会红的那一半」。改成从 knex 的query事件上读真实find()发出去的那条语句。skipIf(!sharedMongod)罩着。于是在拉不到 mongod 二进制的环境里，整个文件 —— 包括唯一能抓住功能被删的那半 —— 全体静默，而且报绿。挪到skipIf外面，用一个只构造、不 connect 的 driver 跑（构造函数不开 socket）。环境限制（需要 CI 覆盖的部分）
MongoDB 的属性用例在本环境跑不了。
mongodb-memory-server要下载的fastdl.mongodb.org被网络策略挡掉（agent proxy 403，__agentproxy/status的recentRelayFailures里有记录），所以需要真实 mongod 的用例是 skipped。上面第 2 条改动的直接结果是：即便如此，4 条 sort-spec 断言仍然真实执行并通过 —— 换成改之前，这个文件在这里会是 17 skipped、0 assertion，看起来跟「全过」一模一样。真实 mongod 上的那一半以 CI 为准。Postgres 的数字是本机真跑的（本地
initdb起的 16.13 实例，2M 行、390 MB，用完已拆）；SQLite 由driver-sql套件覆盖。MySQL 没量。关联
main）。sort/select/expand指向不存在的字段时被静默丢弃（filter 轴已收口，这三条轴还没有） #4226 /sort的点号路径（?sort=account.company_name）仍然静默降级为「不排序」——#4226 收口后唯一漏网的 sort 形态 #4256 —— 「未生效的东西不该看起来生效了」，本 PR 里对应联邦表那条 warn。MongoDBDriver.findOne完全忽略query.orderBy（也忽略query.fields），所以engine.findOne({ orderBy: [...] })在 Mongo 上返回的是任意一条而不是排序后的第一条，与 SQL 分歧。同一族缺陷但不是本 issue，而且本环境跑不了 mongod，盲改比不改更糟。🤖 Generated with Claude Code