Repository navigation
fix(shard-engine): aggregate a bigint column on a soft-delete table - #395
Conversation
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Caution Review failedThe pull request is closed. ℹ️ Recent review info⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (1)
WalkthroughThe shard engine now keeps aggregate companions aligned with soft-delete state and projected values. Aggregate and group-by queries use companions when SQL reduction is unsafe. Documentation covers re-projection migrations and SQL console decoding. ChangesAggregate companion behavior
Estimated code review effort: 4 (Complex) | ~45 minutes Sequence Diagram(s)sequenceDiagram
participant QueryAPI
participant ctxdb
participant SQLScan
participant AggregateCompanion
QueryAPI->>ctxdb: Request aggregate or group-by
ctxdb->>ctxdb: Detect projected grouping and reducer fields
alt SQL can reduce the fields
ctxdb->>SQLScan: Execute SQL reduction
else SQL cannot safely reduce the fields
ctxdb->>AggregateCompanion: Read live companion values
end
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for following the naming conventions! 🙏 |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
Codecov Report✅ All modified and coverable lines are covered by tests. Additional details and impacted files@@ Coverage Diff @@
## alpha #395 +/- ##
==========================================
+ Coverage 87.09% 87.15% +0.06%
==========================================
Files 1172 1181 +9
Lines 63383 64036 +653
Branches 15447 15732 +285
==========================================
+ Hits 55202 55810 +608
- Misses 7654 7700 +46
+ Partials 527 526 -1
🚀 New features to boost your workflow:
|
Merging this PR will improve performance by 10.91%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ⚡ | count, no attributes |
67.1 µs | 60.4 µs | +11% |
| ⚡ | flat 3 primitives (the notify.send attribute shape) |
61.9 µs | 55.8 µs | +10.94% |
| ⚡ | baseline (Object.keys + toInternal + path spread per field) |
72.1 µs | 65.1 µs | +10.81% |
Tip
Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.
Comparing fix/312-bigint-aggregate-gaps (cf6c588) with alpha (2cae0f8)2
Footnotes
-
10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
alpha(9be6dc9) during the generation of this report, so 2cae0f8 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
|
@coderabbitai review |
✅ Action performedReview finished.
|
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/shard-engine/src/ctx-db.ts`:
- Line 3203: Update the aggregate-index condition around scanRefusesAny so the
companion path is used only when planned.partial is empty or fully constrains
every field in planned.index.by. For incomplete group-key filters, add
companion-side filtering or preserve the existing refusal/fallback path,
ensuring queries such as where {a: 1} on an index keyed by [a, b] cannot return
unrelated groups.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 7ad036e6-535b-4214-9488-71a7ffa00cc2
⛔ Files ignored due to path filters (5)
packages/shard-engine/__tests__/ctx-db.bigint-bytes.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/shard-engine/__tests__/ctx-db.soft-delete.companions.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**plans/312-bigint-aggregate-gaps.mdis excluded by none and included by noneplans/315-durable-companion-backfill-marker.mdis excluded by none and included by noneplans/README.mdis excluded by none and included by none
📒 Files selected for processing (7)
apps/docs/src/content/docs/concepts/migrations.mdxpackages/shard-engine/src/aggregate-tally.tspackages/shard-engine/src/ctx-db-backfill.tspackages/shard-engine/src/ctx-db-companions.tspackages/shard-engine/src/ctx-db.tspackages/shard-engine/src/query-args.tspackages/shard-engine/src/sql-console.ts
`groupBy` took the companion whenever an index matched its `by` tuple,
including when the request pinned only PART of that tuple. Neither
companion path can serve that: the single-row lookup needs the whole
tuple, so a partial pin fell to the full walk, which reads every
companion row and returns them all. The `where` was silently dropped —
`by: ["a", "b"]` with `where: { a: 1 }` came back with the `a !== 1`
groups included.
Pre-existing: `collectPartialKey` accepts a subset of the by-fields,
where `planAggregateLookup` (which `aggregate`/`count` use) requires
every one of them pinned, so only `groupBy` had the hole. It was reported
against this branch because the soft-delete gate added here routes more
requests down that path, but a plain table reproduces it on `alpha`.
A partial pin now falls through to the scan, which compiles the predicate
properly. Checked before `ensureBackfilled` so a request that cannot use
the companion does not pay to rebuild it.
Reported by CodeRabbit on #395.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
`recomputeExtreme` answers "the row carrying this group's stored extreme just left, what is the new one?" with `MIN`/`MAX(json_extract(...))`. For a column stored as an order-preserving sort key that reduces the padded KEY, and SQLite coerces the numeric-looking text to a REAL: deleting the max row of a group whose real extreme was 9 wrote ~1e39 into the companion, and every later aggregate read returned it. Silent, and out of reach of the reader's guard. `assertReducibleBySql` refuses such a field on the scan, but this is the write path — refusing here would break `delete`. The group is now folded through `foldAggregateTally`, the same reducer both backfills seed a group with, so a recomputed extreme cannot disagree with a rebuilt one. Gated on `mayHoldProjectedValue` rather than `isProjectedKind`: the projection keys off the RUNTIME value type (`sqlComparableProjection` branches on `typeof value === "bigint"`), so a `v.any()` / `v.union()` / `v.from()` column holding a bigint is stored as the same padded key as a declared one. Over-including is safe here because this branch only reads documents; the reader's guard must stay narrow because it refuses. Not `ORDER BY json_extract(...) LIMIT 1`, which would read one row instead of the group: the key is order-preserving only against other keys, and SQLite orders every TEXT after every numeric — so on a mixed `v.any()` column the extreme would be decided by storage class rather than magnitude. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
A `v.bigint()` / `v.bytes()` column could not be aggregated at all on a `.softDelete()` table, at any magnitude. The readers disqualified the maintained `__agg_` companion whenever the table had a soft-delete mode, because the companion counted deleted rows; that sent every aggregate to the SQL scan, which refuses a projected field outright, since reducing the order-preserving key returns nonsense rather than a value. The companion now tallies LIVE rows only. A soft delete is mechanically a patch that stamps the marker column, so it already reaches `syncAggregates` as an ordinary `(previous, next)` pair — gating both sides on liveness decomposes it into "remove the live row, add nothing", and a restore into its mirror, with no second code path. The record-side predicate sits beside the `softDeleteScope` it mirrors, because a difference between the two is a maintained counter that silently disagrees with the scan it replaces. The 2^53 companion bound is enforced independently of that gate. It is a STORAGE constraint, not a property of today's tally: gating it on liveness let a row that is dead on arrival — or patched while dead — carry a value the REAL `__value__` column cannot hold, with the refusal landing later on `restore()`, a write the caller cannot fix. Worse, such a row is inert only while the schema still declares the marker; dropping `.softDelete()` later makes every row read as live, and the next rebuild folds the out-of-range value and throws on every write path — including the delete that would remove it. The readers take the companion on a soft-delete table only when the SQL scan would refuse (`scanRefusesAny`, derived from the predicate `assertReducibleBySql` refuses on, so gate and assertion cannot drift). Reaching it calls `ensureBackfilled`, whose rebuild is an unconditional TRUNCATE + full scan memoised per ctx-db instance, and codegen builds one per dispatch and again per subscription re-run. Routing every soft-delete aggregate through it would trade one SQL COUNT for a full JS decode of the table plus a companion rewrite, on a read, per request. Plan 314 covers making that marker durable, which retires this gate. No migration: each ctx-db rebuilds a companion from source on first touch, so the format change heals on the next read or write. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
The throw fires on the INSERT, but read "cannot be aggregated exactly", which is a puzzle at the point it lands: nothing in the call mentions an aggregate. Declaring an aggregateIndex over a bigint column is what caps that column, so the message says so, and names dropping the index as one of the ways out. Behaviour is unchanged; the cap stands. Lifting it needs an exact tally — a decimal-string column accumulated with BigInt on the path every insert takes — and `AggregateResult` widened past `null | number` through every adapter, which prices the whole hot write path for summing snowflake ids. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
Plan 303 asked whether the SQL console should wire-decode `__doc__`, since `SELECT *` renders stored tags and 40-character padded sort keys. It should not: this surface shows the DATABASE, and the studio's data browser shows the MODEL and decodes there. Decoding would disagree with the `json_extract(__doc__, …)` the same operator types next, and would change what an existing saved query returns. Comment only. Recorded here so the question is not reopened from the symptom. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
`__lunora_reproject__<table>` is reserved by the framework rather than declared in a project's `lunora/`, and no page mentioned it — so an operator had no way to learn their rows might be affected, which was the real gap behind plan 311's follow-up. Documents what it rewrites, and that `--dry-run` already answers "do I need this?" per shard: the transform returns nothing for a row already in the current projection, and the runner counts only rows it would rewrite, so a dry run's `changed` IS the legacy-row count. `migrate status` cannot answer it — a shard that never ran the backfill has no run-state row. With the caveat that makes `changed: 0` honest: the transform also skips a row it cannot re-project at all (a bigint wider than the 39-digit key, reachable for a uint256-shaped value) and counts it processed, not changed. The only signal is a warning in the shard's logs. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
The spun-out plan was filed as 314, but `alpha` had already taken that number (and 313) while this branch was in flight. Renumbered to 315; comment-only. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
312 — execution record plus two adversarial-review rounds. The reviews found three defects beyond the plan's own two gaps, two of them silent and one introduced by the fix: the padded-key recompute, the same guard keyed off the declared type (missing `v.any()` columns), and the liveness gate bypassing the 2^53 storage cap so a dead-on-arrival row could never be restored — and, after a later `.softDelete()` removal, could brick every write path on the table. Also recorded: WS1's measurement, because it changed the sizing — no first-party schema declares `.softDelete()`, so gap 1 had no in-repo consumer, and §4.1's "it also speeds up every other aggregate" claim is now known to be false. And one defect found, reproducible, and deliberately NOT fixed: `assertReducibleBySql` is narrow the same way the recompute was, so an aggregate over a `v.any()` column holding bigints is neither refused nor answered. Widening it would refuse every aggregate over a wide-kind column including the majority holding plain numbers, so it needs its own plan rather than a rushed guard. 315 — new plan, spun out of 312's review. `ensureBackfilledIndex` is memoised per ctx-db instance and codegen builds one per dispatch, so every table with an aggregateIndex rescans on the first companion touch of every request. Pre-existing on alpha; landing it retires the gate 312 added. 303 closed: the console decision shipped two commits back, and the plan's other half (a `__proto__` field dropped on encode) shipped in f8b05d0. 311's follow-up closed as already-done — `--dry-run` reports the count. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
`groupBy` took the companion whenever an index matched its `by` tuple,
including when the request pinned only PART of that tuple. Neither
companion path can serve that: the single-row lookup needs the whole
tuple, so a partial pin fell to the full walk, which reads every
companion row and returns them all. The `where` was silently dropped —
`by: ["a", "b"]` with `where: { a: 1 }` came back with the `a !== 1`
groups included.
Pre-existing: `collectPartialKey` accepts a subset of the by-fields,
where `planAggregateLookup` (which `aggregate`/`count` use) requires
every one of them pinned, so only `groupBy` had the hole. It was reported
against this branch because the soft-delete gate added here routes more
requests down that path, but a plain table reproduces it on `alpha`.
A partial pin now falls through to the scan, which compiles the predicate
properly. Checked before `ensureBackfilled` so a request that cannot use
the companion does not pay to rebuild it.
Reported by CodeRabbit on #395.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
66e7310 to
cf6c588
Compare
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Closes plan 312's gap 1. The headline is not the gap — it is that closing it surfaced three further defects, two of them silent wrong answers and one introduced by the fix itself, all found by review rather than by the suite.
The gap this closes
A
v.bigint()/v.bytes()column is stored as a fixed-width order-preserving padded TEXT key, which SQL cannot reduce —assertReducibleBySqlrefuses it on the scan. The readers also disqualified the maintained__agg_companion whenever a table had.softDelete(), because the companion counted deleted rows. So such a table always fell to the refusing scan: no aggregate over a bigint column at any magnitude.The companion now tallies live rows only. A soft delete is mechanically a patch that stamps the marker, so it already reaches
syncAggregatesas an ordinary(previous, next)pair — gating both sides on liveness decomposes it into "remove the live row, add nothing", with no second code path and no migration.What review found
recomputeExtremereduced the padded sort KEY in SQL, so deleting the max row of a bigintmaxindex stored ~1e39 as the group's extreme, and every later read returned itv.any()/v.union()/v.from()column holding a bigint took the SQL branch — same wrong number, one declaration awayassertReducibleBySqlis narrow the same way as (2), so an aggregate over av.any()column holding bigints is neither refused nor answered(3) is the one worth reading twice. The cap is a storage constraint that was enforced from the tally branch, so gating the contribution on liveness silently stopped applying it:
And the escalation: that row is inert only while the schema still declares the marker. Drop
.softDelete()later — an ordinary edit — and every row reads as live, the next rebuild folds the out-of-range value, and every write path throws, including the delete that would remove it. Read-alive, write-dead, no in-band repair. The bound is now enforced on the post-image independently of liveness, restoring exactly the set it applied to before.(4) is deliberately left open. Widening the refusal would reject every aggregate over a wide-kind column, including the majority holding plain numbers that work correctly today. Doing it properly needs a probe-vs-refuse-vs-route decision, recorded with its reproduction in
plans/312§12.Scope decision worth reviewing
The readers take the companion on a soft-delete table only when the scan would refuse, not whenever it is available. Reaching it calls
ensureBackfilled, whose rebuild is an unconditional TRUNCATE + full scan memoised per ctx-db instance — and codegen builds one per dispatch, and again per subscription re-run. Routing every soft-delete aggregate through it would trade one SQLCOUNTfor a full JS decode of the table plus a companion rewrite, on a read, per request.That gate is a workaround, and it is labelled as one: plan 315 (new, in this PR) covers making the backfill marker durable, which retires the gate, its note, and the asymmetry between the three readers. That waste is pre-existing on
alphaand affects every table with anaggregateIndex, not just soft-delete ones.Plan 312 §4.1's secondary claim — that this would "also speed up every non-bigint aggregate on a soft-delete table" — is now known to be false, and is recorded as such.
Gap 2: unchanged, still loud
The 2^53 cap on an
aggregateIndexover a bigint column stands. Lifting it needs an exact tally (a decimal-string column accumulated withBigInton the path every insert takes) andAggregateResultwidened pastnull | numberthrough every adapter — priced for "sum a column of snowflake ids", which nobody has asked for. Only the message changed, to name the aggregateIndex as the thing that capped the column, since the throw fires on the insert.Commits
Ordered so each is green on its own.
b110d59dehad to come first: gap 1's fix is what routes a soft-delete bigint aggregate through the companion, which is what exposes the recompute defect — the reverse order leaves the middle commit red.Verification
@lunora/shard-engine1049/1049 ·@lunora/do523/523 ·lint:types,lint:eslint, prettier,lint:package-jsonclean ·api:checkmatches all 47 snapshots (after a fullbuild:packages— a partial build reports drift in packages this branch never touched).Every regression test was verified to fail with its fix reverted, not merely to pass: the recompute test returns
1e+39, and the eager-backfill test returns19instead of10.Note for the reviewer
Rebased onto current
alpha, which had moved 133 commits. That surfaced a collision worth knowing about:alphaalready carries plan 314, "Stop shipping an English NLP library in every Worker" — the same defect a plan in this branch had been filed for independently. That duplicate plan file was deleted rather than merged, and the spin-out renumbered to 315.The deleted plan carried measurements alpha's 314 does not:
redact()costs 5.411 ms/call vs 0.125 ms with the seven NLP-triggering rule keys excluded (43×), the package bundles to 143,735 B gzipped of which ~140 KB iscompromise, andexcludeis not a safe workaround because it drops key-name matching too. Filed upstream as visulima#830, since thecompromiseimport is static and cannot be dropped downstream. Those belong inplans/314-compromise-in-every-worker.mdas a follow-up.🤖 Generated with Claude Code
https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
Summary by CodeRabbit
New Features
Bug Fixes
Documentation