fix(batch,drivers): 分页确定性的两个后续 —— 把「有 gate」变成真的，把批处理翻页改成 keyset (#4363) - #4416
Merged
Conversation
…#4363) Three shared case-sets in `@objectstack/spec/data` -- filter combinator semantics (#3774), temporal storage form (ADR-0053), deterministic paged reads (objectui#3106 / #4363) -- each shipped with some version of the claim that a future driver "is held to this by a gate rather than by remembering it". There was no gate. The case-sets are exports sitting in a package; nothing obliged a driver to import them. Measured on main, the matrix had three holes: driver-sqlite-wasm ran neither pagination case-set, and neither it nor driver-mongodb ran the filter-logic one -- including a hole in the very case-set whose changeset made the claim. That is the declared-not-enforced shape Prime Directive #10 is about, so the fix is the gate rather than a correction to the sentence. scripts/check-driver-conformance.mjs holds three invariants: CONSUMED every (driver x case-set) cell is covered -- a file under the package's src/ imports AND drives the case-set's marker export -- or carries a measured DEBT/EXEMPT entry. CLASSIFIED every case-set exported by spec/src/data/*-conformance.ts is named in the requirement table, so a new shared fixture nobody classified fails rather than silently dropping out of coverage (#4203). It caught an unclassified TEMPORAL_TIME_CASES on its first run. RECONCILED in both directions -- a ledger row for a cell that is now covered, or for a driver/case-set that no longer exists, is an error. Both axes are discovered from disk, never listed, so a new driver package is in scope the moment it exists. Coverage requires the marker to be referenced outside its own import: an unused import is not coverage, and a locally re-declared fixture is not the shared standard. --self-test drives all three distinctions plus both discoveries. driver-sqlite-wasm gains the pagination suite the gate found missing. It inherits SqlDriver's ORDER BY construction, so nothing is re-implemented; what the suite pins is that the clause survives a different ENGINE, since this driver swaps knex's transport for a custom sql.js dialect that compiles, executes and marshals every row through its own path. The two filter-logic holes are ledgered as DEBT with their reasons printed on every run and tracked in #4405, rather than fixed here. The mongodb row is the substantive one: translateFilter is an independent FilterCondition backend, the fifth, and the one #3774 never enrolled when it counted "the four". Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016niiseyRn43L9M4JdAwuqD
…op skipping rows (#4363) #4363 made a single paged read a partition of its result set. It could not make a WALK one: seven background scans paged with a growing `offset` while writing to the very rows they were reading, and an offset counts into a set those writes are changing. Rows slide past the cursor and are never visited. In none of these is that a slow page -- it is a wrong answer wearing the shape of a clean run: rebuildApproverIndex built its desired state by walking sys_approval_request WHERE status='pending' with no orderBy at all, then DELETED every index row that state did not explain. A skipped request meant an approver silently dropped from someone's queue. (The loop beside it ordered by created_at, not unique, so its pages were never a partition either.) verifyFileReferences decides which files nothing references. A record it never visits is reported as unreferenced. backfillFileReferences and the pinyin companion backfill rewrite each row they read, so their own writes shifted the set out from under the cursor. scanValueShapes vouches that no stored value is off-shape, and opens a migration gate on that evidence. All now go through `keysetWalk` (@objectstack/types): order by a unique key, seek past the last one. A row's key does not move when the row is updated and cannot be shifted when another is deleted, so the walk is stable under exactly the mutation these functions perform. It is also O(n) rather than O(n^2/page) -- measured on Postgres over 2M rows, deep pages cost ~1.1s by offset against ~0.09s by seek. Three details the helper owns so six copies cannot drift on them: - the cursor merges with `$and` rather than being spread into the caller's filter, so a walk whose own `where` constrains the key column keeps that constraint instead of having it overwritten; - with a `max` cap it reads one row past the cap to tell "the cap stopped us" from "the source ended exactly there", so a walk that read everything does not report `truncated`; - a cursor that fails to advance stops the walk instead of spinning. A reader that drops the seek would otherwise hang, and a hang is the one failure nobody can read off a log. Found by a test double doing exactly that. One deliberate non-conversion: the REST export stream keeps its offset. It honors a caller-chosen sort, and seeking would mean re-ordering the export by id -- changing what the user asked for to fix a cost. Its pages are already a partition since #4363; only the depth cost remains. The storage and pinyin fakes now THROW on an offset instead of serving one, so the conversion is pinned rather than merely passing. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_016niiseyRn43L9M4JdAwuqD
…-breaker-missing-p98qf0
|
The latest updates on your projects. Learn more about Vercel for GitHub. 1 Skipped Deployment
|
github-actions
Bot
added
documentation
os-zhuang
marked this pull request as ready for review
July 31, 2026 12:57
Contributor
📓 Docs Drift CheckThis PR changes 5 package(s): 19 hand-written doc(s) reference the affected code and may need an implementation-accuracy re-verification:
|
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
2 participants
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
#4378(#4363)的两个后续。两个 commit 各自独立、可分开 review 也可分开 revert，放在一起只是因为本任务只有一条分支。
bfaa13a— driver conformance gate（+ 它当场找到的 wasm 分页缺口）106a300— 批处理翻页改 keyset一、
bfaa13a：那句「held to by a gate」以前是假的packages/spec/src/data/*-conformance.ts里的共享用例集存在的理由，是让各自独立实现的 driver 回答同一个标准：filter 组合子语义（#3774）、temporal 存储形态（ADR-0053）、分页确定性（objectui#3106 / #4363）。三份 changeset 各自都写了某种版本的「a future driver is held to this by a gate rather than by remembering it」。没有这个 gate。 用例集只是躺在包里的导出，没有任何东西逼一个 driver 去 import 它。在
main上实测，矩阵有三个洞 —— 其中一个就在那句话所属的用例集里：这是 Prime Directive #10 的 declared ≠ enforced 形态，所以修法是把 gate 做出来，而不是发一条更正把那句话删掉。
scripts/check-driver-conformance.mjs三条不变量：
src/下有文件 import 且真的用了该用例集的 marker 导出 —— 要么带一条量过的 DEBT/EXEMPT 台账条目spec/src/data/*-conformance.ts导出的每个用例集都必须在需求表里有名字。新加的共享 fixture 没人分类就挂，而不是悄悄掉出覆盖（#4203 那条教训，用在真正会烂的那个方向）。它第一次跑就抓到了没分类的TEMPORAL_TIME_CASES两条轴都从磁盘发现，不写死列表 —— 新 driver 包一存在就在范围内。「覆盖」要求 marker 在自己的 import 之外被引用过：没用到的 import 不算覆盖，本地重新声明一份同名 fixture 也不算共享标准。
--self-test把这三种区分和两条发现都跑一遍。范围是刻意划的：只管
packages/plugins/driver-*。packages/formula和 service-analytics 也消费同一批用例集，但它们不是 driver、实现的是不同子集，把它们收进来就得逐个回答「哪些用例集对你适用」——那是这个 gate 只能猜的问题。写在脚本头注里了。顺手补上它找到的一个洞
driver-sqlite-wasm拿到分页 conformance 套件（17 个用例）。它继承SqlDriver的 ORDER BY 构造，所以什么都没重新实现 —— 套件钉的是那条子句能不能活过另一个引擎：这个 driver 把 knex 的传输层换成了自定义 sql.js 方言，语句的编译、执行、每一行的 marshal 都走它自己的路径，一个把末尾ORDER BY id重排或丢掉的方言会在别处任何套件里都不报错。这和隔壁 temporal 套件存在的理由是同一个。两个 filter-logic 洞记账而不是就地修
进 DEBT 台账，理由每次运行都打印出来，跟踪在 #4405。mongodb 那条是实质性的：
translateFilter是一个独立的 FilterCondition backend —— 第五个，也是 #3774 数「四个」时没数进去的那个。二、
106a300：批处理翻页改 keyset#4363 让一次分页读取成为结果集的划分。它没法让一次遍历成为划分：七处后台扫描一边用递增
offset翻页，一边写它正在读的那些行 —— 而 offset 是往一个正被自己的写改变的集合里数数。行从游标底下滑过去，永远不会被访问。这在每一处都不是「慢一页」，而是穿着干净外衣的错误答案：
rebuildApproverIndexsys_approval_request WHERE status='pending'，完全没有 orderBy，然后按这份「期望状态」删除解释不了的索引行。漏一个请求 = 某人的待办审批被静默删掉。（紧挨着的第二个循环按created_at排 —— 不唯一，所以它的页也从来不是划分。）verifyFileReferencesbackfillFileReferences/ 拼音 companion 回填scanValueShapes全部改走
keysetWalk（@objectstack/types）：按唯一键排序，seek 过上一个键而不是从头数。行被更新时键不动，别的行被删时键也不会被挪，所以遍历对这些函数正在做的那种改动是稳定的。顺带从 O(n²/page) 变成 O(n) —— Postgres 2M 行实测，深页 offset ~1.1 s、seek ~0.09 s。三个细节由 helper 独占，免得六份拷贝各自漂移：
$and合并而不是展开进调用方的 filter —— 调用方自己的where如果也约束了键列（{ id: { $in: [...] } }），不会被游标静默覆盖掉。max上限时多读一行，用来区分「是上限拦住的」和「数据刚好在这里没了」。没有这一行，读完了整张表的遍历照样报truncated，而据此行动的调用方会去找从来没被扣下的行。一处刻意不改：REST 的导出流保留 offset。它要尊重调用方选的排序，而 keyset 就得把导出重排成
id序 —— 为了省成本去改用户要的东西。它的页自 #4363 起本来就是划分了，剩下的只是深度成本。storage 和 pinyin 的 fake engine 现在遇到
offset直接抛而不是照做，所以这次转换是被钉住的，不是碰巧过。验证
合并
main（到302e972）后pnpm install --frozen-lockfile+ 全量pnpm build+ 清.objectstack/，全部重跑：pnpm test@objectstack/typesdriver-sqlite-wasmobjectql/plugin-approvals/service-storage/plugin-pinyin-searchpnpm check:driver-conformance--self-test通过check:type-check-coverage/check:doc-authoring/check:role-word/check:nul-bytestypecheck（types）+ 改动文件 eslintgate 的三条不变量都做过变异验证（拆了会红，不是跑绿了）：删掉 wasm 的分页套件 → 报 CONSUMED（2 条）；把台账条目指到一个已覆盖的格子 → 报 RECONCILED；
TEMPORAL_TIME_CASES没分类 → 报 CLASSIFIED（这条是真实发生的，不是构造的）。service-storage的tsc --noEmit错误数改动前后都是 42，没有新增（该包在覆盖率台账里，这些是冻结债务）。关联
orderByis a partition too — the shape every list view actually sends (#4363) #4378 —— 本 PR 的来源。MongoDBDriver.findOne忽略orderBy，仍未修。$orbranch contents are OR-ed instead of AND-ed, widening every$orfilter #3774 / ADR-0053 / objectui#3106 —— 三个共享用例集的出处，也是那句「有 gate」的三个出处。🤖 Generated with Claude Code
Generated by Claude Code