Repository navigation
fix(shard-engine): size emitted SQL to workerd's SQLite limits - #396
Conversation
Workerd's SQLite build — the one behind both Durable Object storage and D1 — caps compound SELECT terms at 5 (stock: 500), bound parameters at 100 (stock: 500,000), and LIKE patterns at 50 bytes (stock: 50,000). Over any of them the statement fails at runtime with SQLITE_ERROR. Three families of generated statement were over the line, and none of it reproduces under node:sqlite, which runs the stock limits. Compound SELECT: the by-id probe unions one branch per non-global table, so a bare `ctx.db.delete(id)` / `ctx.db.patch(id, …)` threw "too many terms in compound SELECT" on any schema wider than five tables (#385). `unionAll` nests the branches five at a time — a sub-select restarts the parser's term counter, so it stays one round-trip and returns the same rows at any branch count. The FTS vocabulary scorer, whose branches scale with query terms, goes through it too. Bound parameters: `IN (…)` bound one placeholder per item, and the relation semijoin deliberately resolves up to 5,000 join keys; the batched id→table probe budgeted against 999; bulk INSERT bound three per row against a 500-row batch cap; the global search companion inserted 50 rows (150 parameters) per statement. `sqliteInList` renders a long list as a single `json_each(?)` parameter — same membership, placeholder count independent of set size — keeping short lists literal so an index still plans against them. INSERTs chunk at 32 rows. LIKE patterns: `contains` compiled to `LIKE '%' || term || '%'`, so a filter term over ~48 characters failed the query outright, as did a data-browser filter or a 50-character search term. It now compiles to a position function per dialect — `instr(lower(x), lower(term))` on SQLite, `LOCATE` on MySQL, `strpos` on Postgres — each folding case the way that dialect's LIKE did, so results are unchanged. The wildcard escaping goes away with it: a position test takes the term literally, so a client-supplied `%` can no longer become a live pattern. The caps cannot be exercised locally, so the new suite asserts them on the rendered SQL (compound width, placeholder count, absence of a LIKE pattern) and pairs each with a behavioural case proving the reshaped statement still returns the same rows on a real SQLite build. Closes #385 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Thank you for following the naming conventions! 🙏 |
|
Warning Review limit reached
Next review available in: 33 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: 📒 Files selected for processing (1)
WalkthroughThe PR adds shared SQLite helpers for bounded ChangesSQLite query limit handling
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant ShardEngine
participant SQLHelpers
participant SQLite
ShardEngine->>SQLHelpers: Build bounded ID filter with sqliteInList
ShardEngine->>SQLHelpers: Combine schema probes with unionAll
SQLHelpers->>SQLite: Execute nested UNION queries
SQLite-->>ShardEngine: Return resolved ID-to-table rows
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
There was a problem hiding this comment.
Actionable comments posted: 1
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/shard-engine/src/ctx-db.ts (1)
2417-2456: 🩺 Stability & Availability | 🟠 Major | 🏗️ Heavy liftSplit the table probe when the schema has more than 100 tables.
Line 2443 reduces the per-branch list budget, but it does not bound total parameters. With 101 non-global tables, one ID creates 101 literal parameters. With two or more IDs, each branch uses one
json_eachparameter, which still creates 101 parameters. Workerd rejects both statements.Partition
nonGlobalTablesinto groups that fit the variable limit. Run one nested union per group and merge the rows intoresolved.unionAllonly fixes the compound-select limit.🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/shard-engine/src/ctx-db.ts` around lines 2417 - 2456, The locateTablesByIds probe must split nonGlobalTables into groups whose total SQL parameters stay within Workerd’s 100-variable limit; per-branch budgeting alone still exceeds the cap for wide schemas. Partition nonGlobalTables into parameter-safe groups, execute a separate unionAll probe for each group, and merge each result into resolved, preserving the existing ID filtering and table-name mapping.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/shard-engine/src/drizzle.ts`:
- Around line 45-46: Update isJsonSafe to accept numbers only when
Number.isFinite(value) is true, while continuing to accept null, booleans, and
strings. Ensure NaN, Infinity, and -Infinity use the existing fallback literal
path so list membership semantics are preserved.
---
Outside diff comments:
In `@packages/shard-engine/src/ctx-db.ts`:
- Around line 2417-2456: The locateTablesByIds probe must split nonGlobalTables
into groups whose total SQL parameters stay within Workerd’s 100-variable limit;
per-branch budgeting alone still exceeds the cap for wide schemas. Partition
nonGlobalTables into parameter-safe groups, execute a separate unionAll probe
for each group, and merge each result into resolved, preserving the existing ID
filtering and table-name mapping.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: c70b4da8-d522-4381-a181-85340616bd21
⛔ Files ignored due to path filters (4)
api-snapshots/shard-engine.api.mdis excluded by none and included by nonepackages/shard-engine/__tests__/ctx-db.by-id-table-scope.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/shard-engine/__tests__/where-sql.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/shard-engine/__tests__/workerd-sql-limits.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (9)
packages/shard-engine/src/ctx-db-rank-page.tspackages/shard-engine/src/ctx-db-shapes.tspackages/shard-engine/src/ctx-db.tspackages/shard-engine/src/drizzle.tspackages/shard-engine/src/index.tspackages/shard-engine/src/introspect.tspackages/shard-engine/src/where-sql.tspackages/sql-store/src/ctx-db.tspackages/sql-store/src/search-layout.ts
Merging this PR will regress 1 benchmark
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | flat where on a relation-bearing table (no relation predicate) |
16.6 ms | 18.6 ms | -10.72% |
| ⚡ | semijoin: reactions where message.is(author.is(broad)) |
95 ms | 33.2 ms | ×2.9 |
| ⚡ | scan: count whole table (SELECT COUNT(*)) |
230.9 µs | 203.6 µs | +13.41% |
| ⚡ | findMany — plain table (no scope) |
93.3 ms | 82.6 ms | +13.01% |
| ⚡ | scan: count whole table (SELECT COUNT(*)) |
247.3 µs | 219.6 µs | +12.6% |
| ⚡ | flat 3 primitives (the notify.send attribute shape) |
61.9 µs | 55.6 µs | +11.42% |
| ⚡ | count, no attributes |
67.1 µs | 60.2 µs | +11.39% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing fix/385-workerd-sql-limits (ef7fab1) with alpha (2cae0f8)2
Footnotes
-
10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
alpha(277685d) during the generation of this report, so 2cae0f8 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## alpha #396 +/- ##
==========================================
+ Coverage 87.09% 87.16% +0.07%
==========================================
Files 1172 1181 +9
Lines 63383 64100 +717
Branches 15447 15772 +325
==========================================
+ Hits 55202 55874 +672
- Misses 7654 7700 +46
+ Partials 527 526 -1
🚀 New features to boost your workflow:
|
Nesting the union branches lifted the compound-SELECT ceiling but not the bound-parameter one: every branch still spends at least one placeholder — the id in `locateRowById`, the `json_each` list in `locateTablesByIds` — so a schema past 100 non-global tables bound more than Workerd allows and the probe failed to prepare. Both probes now run a chunk of tables at a time, and the per-branch list budget divides the cap within a chunk. `sqliteInList` also treated any number as JSON-safe, but `JSON.stringify` writes NaN and ±Infinity as null — a long list holding one would have stopped matching it and started matching null. Non-finite numbers keep the literal form. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The per-list threshold only bounded one list: three 40-item `in` filters in one `where` are ordinary app code and bound 120 placeholders while each list sat "within budget" on its own. The compiler now counts the lists in the tree and splits one statement-wide budget between them. A list JSON cannot carry — bytes, a non-finite number, malformed text — had no bounded form and silently fell back to a placeholder per item, so an over-budget one emitted SQL certain to fail to prepare. It now throws with a message that says which limit it hit. Lone surrogates join the excluded set: JSON folds them to U+FFFD, so the same list matched different rows either side of the threshold. `inList` defaulted to the unbounded literal form while its sibling `containsExpr` defaulted to the safe one, so every SQLite strategy had to remember to opt in and the next one to forget would fail only in production. Flipped: the bounded form is the default and the dialects without `json_each` opt out. Revert the search companion's prefix term to `LIKE` and its insert chunk to 50. That layout is chosen only when a dialect has no FTS5, which is Postgres and MySQL alone — the cap used to justify both changes cannot apply there, and on Postgres the range was strictly worse: the token btree declares `text_pattern_ops` precisely so a prefix `LIKE` stays indexed, an operator class that cannot answer `>=` at all, and a collation-ordered range is a different set from a character-wise `LIKE`. Covered by the multi-token search test the suite never had. Close the same LIKE cap in the request log, which runs on Durable Object SQLite and built a pattern from a caller-supplied function-path prefix — 54 bytes for a realistic path, against a cap of 50. The limits themselves were spelled out as five hand-derived constants with nothing tying them to the limit they came from; they now read from one record. A real-workerd suite pins all three caps and the `json_each` authorization the bounded list form rests on — the sibling suite runs on node:sqlite, which has none of these limits and passes either way. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
There was a problem hiding this comment.
Caution
Some comments are outside the diff and can’t be posted inline due to platform limitations.
⚠️ Outside diff range comments (1)
packages/shard-engine/src/where-sql.ts (1)
278-332: 🎯 Functional Correctness | 🟠 Major | 🏗️ Heavy liftAccount for all statement parameters.
WHERE_LIST_PARAM_BUDGETonly limits list placeholders. It does not count scalar predicates, cursor predicates, or other bound expressions in the same statement.For example, two 25-item lists use 50 placeholders. Fifty-one
eqpredicates then produce at least 101 placeholders before any cursor or limit bindings. Workerd rejects that statement.Track one parameter budget across every bound expression, or reserve the non-list parameters before splitting the list budget.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the rest with a brief reason, keep changes minimal, and validate. In `@packages/shard-engine/src/where-sql.ts` around lines 278 - 332, Update compileWhereSql and its budget calculation so list placeholders share the complete statement parameter budget with scalar predicates, cursor predicates, and other bound expressions. Count or reserve non-list parameters before deriving the per-list allowance, then pass the remaining budget through the inList strategy while preserving existing empty and zero-list behavior.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Outside diff comments:
In `@packages/shard-engine/src/where-sql.ts`:
- Around line 278-332: Update compileWhereSql and its budget calculation so list
placeholders share the complete statement parameter budget with scalar
predicates, cursor predicates, and other bound expressions. Count or reserve
non-list parameters before deriving the per-list allowance, then pass the
remaining budget through the inList strategy while preserving existing empty and
zero-list behavior.
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 76c8adb7-071f-43ae-973f-469ab96b7302
⛔ Files ignored due to path filters (5)
api-snapshots/shard-engine.api.mdis excluded by none and included by nonepackages/do/__tests__/workerd/sql-limits.workerd.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/shard-engine/__tests__/ctx-db.by-id-table-scope.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/shard-engine/__tests__/workerd-sql-limits.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/sql-store/__tests__/ctx-db-search.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (11)
packages/observability/src/request-log.tspackages/server/src/list-args.tspackages/shard-engine/src/ctx-db-companions.tspackages/shard-engine/src/ctx-db-shapes.tspackages/shard-engine/src/ctx-db.tspackages/shard-engine/src/drizzle.tspackages/shard-engine/src/index.tspackages/shard-engine/src/introspect.tspackages/shard-engine/src/where-sql.tspackages/sql-store/src/ctx-db.tspackages/sql-store/src/search-layout.ts
🚧 Files skipped from review as they are similar to previous changes (5)
- packages/shard-engine/src/introspect.ts
- packages/shard-engine/src/ctx-db.ts
- packages/sql-store/src/ctx-db.ts
- packages/shard-engine/src/drizzle.ts
- packages/shard-engine/src/ctx-db-shapes.ts
Two eslint failures from the previous commit, both in `@lunora/sql-store`:
- `literalInList` was inserted out of alphabetical order in the
`@lunora/shard-engine` import block (`simple-import-sort/imports`).
- The prefix-term assertion sorted a result it had already asserted holds one
element, tripping `unicorn/no-array-sort` and `sonarjs/no-alphabetical-sort`
at once. A single-element array has no order to normalize, so the sort is
gone rather than replaced with a comparator.
While there: that test's comment said the prefix term "matches both of its
documents" while asserting one. It means both of the document's TOKENS
("wonderful", "world"); say so, and note why "wandering" is correctly outside
the `wo` range.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
|
Thank you!! |
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Fixes #385, plus three more limits from the same source that the audit it prompted turned up.
The problem
Workerd's SQLite build — the one behind both Durable Object storage and D1 — sets several limits far below stock SQLite. Over any of them, the statement fails at runtime with
SQLITE_ERROR:COMPOUND_SELECTVARIABLE_NUMBERIN (…)lists, bulkINSERTLIKE_PATTERN_LENGTHcontainsfiltersEXPR_DEPTHMAX_SEARCH_TERMS = 16None of it reproduces locally:
node:sqliteruns the stock limits and exposes no way to lower them.The fixes
Compound SELECT (#385). The bare-id probe unions one branch per non-global table, so
ctx.db.delete(id)/ctx.db.patch(id, …)without anexpectedTablethrewtoo many terms in compound SELECTon any schema wider than five tables — a five-table schema sits exactly at the cap.unionAllnests the branches five at a time: a sub-select restarts the parser's term counter, so it stays one round-trip and returns the same rows at any branch count. The FTS vocabulary scorer, whose branches scale with query terms, goes through it too.Bound parameters.
where: { id: { in: [...] } }bound one placeholder per item, and the relation semijoin deliberately resolves up to 5,000 join keys; the batched id→table probe budgeted against 999; bulkINSERTbound three per row against a 500-row batch cap; the global search companion inserted 50 rows (150 parameters) per statement.sqliteInListrenders a long list as a singlejson_each(?)parameter — same membership, placeholder count independent of set size — and keeps short lists literal so an index still plans against them.INSERTs chunk at 32 rows.LIKE patterns.
containscompiled toLIKE '%' || term || '%', so a filter term over ~48 characters failed the query outright, as did a Studio data-browser filter or a 50-character search term on D1. It now compiles to a position function per dialect —instr(lower(x), lower(term))on SQLite,LOCATEon MySQL,strposon Postgres — each folding case the way that dialect'sLIKEdid, so results are unchanged. The wildcard escaping goes away with it: a position test takes the term literally, so a client-supplied%can no longer become a live pattern.What is not in here
The issue also suggested encoding the table into the id so
asIdscopes at runtime, or removing the bare-id overloads. Neither is here — the probe now works at any table count, and the IDOR concern is already handled byexpectedTableon the per-table facade. ThatasIdreads like it scopes when it doesn't is worth revisiting separately.Testing
The caps can't be exercised locally, so
packages/shard-engine/__tests__/workerd-sql-limits.test.tsasserts each on the rendered SQL — compound width per nesting depth, placeholder count, absence of a LIKE pattern — and pairs each with a behavioural case proving the reshaped statement returns the same rows on a real SQLite build. The compound-width assertions were confirmed to fail without the nesting, and the batched probe was confirmed to actually be reached.sqlite3supports.limit compound_select 5, which is how the 5-vs-6-term boundary and the sub-select reset were verified by hand.🤖 Generated with Claude Code
https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
Summary by CodeRabbit
Bug Fixes
Improvements