Repository navigation
fix(shard-engine): make v.bigint() columns queryable, not just storable - #365
Conversation
The doc-blob codec stored a `v.bigint()` as the wire codec's tagged ARRAY
(`["$lunora.wire$","bigint","10"]`), but every read of a field goes through
`json_extract(__doc__, '$.field')`, and the query side binds its comparison
value through `serializeSqlValue`, which rendered the decimal string "10".
The two could never match. Measured against a real SQLite engine, with three
rows of 9/10/200:
filter eq 10n -> [] (expected ["s10"])
filter gt 9n -> all three rows (expected 10 and 200)
order by asc -> 10, 200, 9
aggregate sum -> null / 0
aggregate max -> the raw string '["$lunora.wire$","bigint","9"]'
count -> 0
Storage worked, so the insert that used to throw now succeeded and a balance
check read a confident, wrong zero. Separately, `estimateBytes` still threw on
a bigint and returned `undefined`, which `recordWrite` converts to a hard
BAD_REQUEST — so with a transaction meter attached, i.e. in production, the
write did not merely query wrong, it failed outright.
Encoding decision: STORE THE COMPARABLE SCALAR, do not teach the query side the
tagged form. `encodeDocJson` now projects top-level bigint/bytes fields to a
JSON number / base64 string and parks the wire-tagged originals under a reserved
`__sql__` key; `serializeSqlValue` binds that same projection. Teaching
`serializeSqlValue` the tagged form was the smaller diff but fixes only
equality — a tagged array is still text to SQLite, so ranges, ORDER BY, SUM and
MIN/MAX stay wrong, and wrong silently. The projection makes `json_extract`
return an INTEGER, so all of them are numeric. The parked original is what keeps
the round-trip exact, which `Number(9007199254740993n)` is not; only the SQL
comparison is approximate above 2^53, which is SQLite's own ceiling. Nested
bigint/bytes keep the wire-tagged form — SQL never addresses them.
Backward compatible in both directions: a doc with no bigint/bytes gains no
reserved key and encodes byte-identically (the OCC compare-and-swap depends on
it), a pre-codec plain-JSON row decodes unchanged, and a row written while
bigints were tagged in place still decodes to a bigint.
Also live and fixed:
- `coerceAggregateNumber` refused bigints, so the maintained companion silently
contributed 0 to a sum over a money column.
- `encodeAggregateKey` threw out of `JSON.stringify` on a bigint `by` field.
- `estimateBytes` threw on bigint (above) and charged an ArrayBuffer as `{}` —
2 characters for a kilobyte.
- The `cdcSync` admin egress returned DECODED post-images through
`jsonResponse`, which is plain `Response.json`: a bigint threw a raw TypeError
and an ArrayBuffer would have reached a consumer's backup as `{}`. It now
wire-encodes, and `applyCdc` decodes, so export -> replay round-trips.
- `encodeDocJson` re-throws the wire codec's bare TypeError/RangeError as a
typed BAD_REQUEST instead of an opaque redacted RPC_FAILED.
- The data browser drops the reserved key rather than showing it as a column.
Audited and NOT live: the sql-store `sqliteDecode` bytes branch already exists
and already normalises a pooled view correctly. Byte identity is genuinely false
for array-position `undefined`, NaN, Infinity and Date, but each of those now
round-trips where it previously degraded to null / an ISO string, so the
docblock claim was corrected rather than the behaviour. A `toJSON`-bearing class
instance in a `v.any()` field still cannot be stored — restoring that would mean
changing `shared/wire-codec.ts`, which is a wire-protocol surface; it is now a
typed error naming the value.
Tests: `ctx-db.bigint-bytes.test.ts` now filters, indexes, orders, counts and
aggregates on a bigint column, on a table mirroring `@lunora/payment`'s
`paymentSessions` money columns — the absence of exactly that coverage is what
let this ship. 14 of its 28 cases fail against the previous code; the CDC case
fails with the literal `TypeError: Do not know how to serialize a BigInt`.
Write-throughput benches unchanged (insert 135.3k -> 139.3k hz, patch 56.1k ->
56.0k hz, both inside rme).
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
The `DOC_ORIGINALS_KEY` docblock claimed `__`-prefixed names are reserved by a
codegen column guard. No such guard exists. The only reserved-name enforcement
in the stack is `RESERVED_TABLE_NAMES` (codegen/src/discover-schema.ts:64),
which covers TABLE names colliding with `ctx.db` members, and
`SYSTEM_INDEX_FIELDS` (server/src/schema.ts:861), which is the two-entry list
`["_creationTime", "_id"]` of indexable system fields. There is no
`startsWith("__")` check, no `/^__/`, and no reserved-field list anywhere in
codegen or server.
So a user field named `__sql__` was silently destroyed, on write, two different
ways — the class of defect this whole change exists to remove:
- With a bigint present, `projected[DOC_ORIGINALS_KEY] = originals` overwrote
the user's value; `{ amount: 10n, __sql__: { keep: "me" } }` read back as
`{ amount: 10n }`, the field gone for good.
- With no bigint at all — nothing projected, so the key should have been inert —
`decodeDocJson` still saw `originals !== undefined` and spread the user's own
object up to the top level, dropping `__sql__`. A document that never went
near a bigint was corrupted on read.
Both writes succeeded. No throw, no warning.
`encodeDocJson` now rejects the name with a typed BAD_REQUEST. That is the
single chokepoint every write takes (insert, insertMany, patch, replace, soft
delete, CDC append), and it is also what makes `decodeDocJson`'s un-projection
unambiguous: a row carrying a user's `__sql__` can never have been stored. A
field-name guard at schema-definition time was the alternative; it is the larger
change and would have to run on the runtime insert path anyway, which is exactly
where this sits.
The docblock now states what is actually true, with the file:line for both
guards that do exist.
Also pins the migration answer for rows written between ab0afaf and this
branch, which hold a bigint tagged in place rather than projected: they read
correctly but `json_extract` comparisons miss them, and any write through the
store re-projects the whole document. The new test asserts both halves — invisible
to `filter` before a `patch`, found after, with the stored text confirmed to be a
plain number. Rows nobody rewrites need a backfill.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Warning Review limit reached
Next review available in: 23 minutes You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (3)
WalkthroughThe PR adds wire encoding for admin RPCs and cursors, SQL-comparable projections for bigint and binary values, document original-value preservation, aggregate validation, and legacy-row reprojection migrations. ChangesWire-safe values and SQL projections
Estimated code review effort: 4 (Complex) | ~60 minutes Sequence Diagram(s)sequenceDiagram
participant AdminClient
participant ShardDO
participant decodeWire
participant AdminOperation
AdminClient->>ShardDO: Send encoded admin arguments
ShardDO->>decodeWire: Decode arguments
decodeWire-->>ShardDO: Return decoded arguments or BAD_REQUEST
ShardDO->>AdminOperation: Dispatch operation
AdminOperation-->>ShardDO: Return operation result
ShardDO-->>AdminClient: Send encodeWire response envelope
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for following the naming conventions! 🙏 |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
Codecov Report❌ Patch coverage is Additional details and impacted files@@ Coverage Diff @@
## alpha #365 +/- ##
========================================
Coverage ? 87.09%
========================================
Files ? 1172
Lines ? 63386
Branches ? 15475
========================================
Hits ? 55204
Misses ? 7663
Partials ? 519
🚀 New features to boost your workflow:
|
Merging this PR will regress 1 benchmark
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | findMany — plain table (no scope) |
77.8 ms | 92.4 ms | -15.78% |
| ⚡ | indexed: groupBy(projectId) — count (read every counter row) |
831.3 µs | 421.6 µs | +97.18% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing fix/265-wire-codec-query-parity (2dfa920) with alpha (4361cbc)2
Footnotes
-
10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
alpha(1ca3f16) during the generation of this report, so 4361cbc was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
`Number(value)` was the wrong projection, and wrong in the one direction that
is never acceptable: it returned rows that do not match.
`Number` collapses every bigint past 2^53 onto the nearest double, so
9007199254740991n / …992n / …993n all become 9007199254740992. Measured:
where { externalId: …993n } -> 2 extra rows (false positives)
unique index, …789n + …790n -> second insert throws ConflictError
.paginate() / orderBy+limit -> raw TypeError out of the cursor encoder
`origin/alpha` returns [] for all three — uniformly wrong, but in the safe
direction. Trading false negatives for false positives is a regression even
though more queries "work": a `where` is routinely an authorization predicate,
and snowflake ids / epoch-nanos / content hashes are exactly why people reach
for v.bigint(). The unique-index case is worse still — data that persists today
stops persisting, under an error claiming "duplicate" about two different values.
The ceiling comment claiming "SQLite's own numeric stack cannot do better" was
false; SQLite carries exact int64 through json_extract. The lossy step was JS.
## Encoding, decided against the rules in order
1. A query must never return a row that does not match.
2. Exactness beats operation coverage; what cannot be exact fails loudly.
3. Keep SUM/MIN/MAX if possible, but not by accepting a wrong `=`.
A bigint now stores as a fixed-width, sign-aware, order-preserving decimal key:
sign character, then the magnitude zero-padded to 39 digits, nines-complemented
under a lower sign character when negative. Verified against a real SQLite
build: `=` is exact at 2^53 and either side of it, ORDER BY is numeric across a
22-digit range including negatives, and ranges narrow correctly. 39 digits
covers the whole unsigned 128-bit range; past it the write is refused rather
than silently mis-sorted.
Plain decimal text was rejected — exact for `=` but sorts "9" after "10", so
ranges and ORDER BY would return wrong rows instead. Raw JSON integer literals
were rejected too, and are a trap worth recording: SQLite does hold exact int64,
but reading such a row back throws `RangeError: Value is too large to be
represented as a JavaScript number` out of the driver, so the row becomes
unreadable, and the comparison side cannot bind an exact int64 either. Checked
on node:sqlite; NOT checked on workerd's SqlStorage, which no longer matters
since neither candidate survived.
Rule 3 is the cost: padded text is not something SQLite can add up. The
aggregate reader now refuses a v.bigint() field on the scan path rather than
return the 1.5e40 that falls out of coercing it, and the maintained companion
refuses a value its REAL column cannot hold exactly instead of folding a rounded
one into an unauditable total. Declaring an aggregateIndex is the supported way
to sum a bigint column.
## Also fixed
- `.paginate()` / orderBy+limit threw out of `encodeCursor`'s bare
JSON.stringify — the same defect class this plan is named after, surviving in
the cursor encoder. Now wire-encoded, and decodeCursor normalises the
RangeError to the existing typed 400.
- `exportShard` was the un-fixed twin of the cdcSync fix: a bigint column 500'd
a whole shard export, a bytes column reached a backup as {}. Fixed at the
boundary instead of per-op — one `adminResponse` encode covers every admin
handler, one `decodeAdminArgs` covers every ingress (so importShard gets what
applyCdc had), and the 12-line inline comment is gone. `createShardClient`
already decodeWires every admin result, and both codecs are the identity on
pure JSON.
- The `__sql__` docblock claimed a historical invariant it cannot provide. The
guard is prospective and now says so; decode additionally claims the key only
when its value is a plain object, so a pre-existing scalar is left alone
rather than exploded into {"0":"h","1":"e",…}. Renamed to `__originals__`,
which is what it holds.
- The Studio browser was showing a plausible-but-wrong integer where alpha
showed an obviously-broken tagged array. It now shows the parked original —
exact, and already JSON-safe for that read path.
- A bigint past Number.MAX_VALUE projected to Infinity, which is !== undefined,
so it counted as projected and then got re-tagged by encodeWire. The width
guard removes the path.
## Structural
The three-branch projection was written twice under a comment asking the copies
to agree. It is now one function (`sql-projection.ts`) that both encodeDocJson
and serializeSqlValue call — the two sides of every comparison are its two call
sites. `??` not `||` at the fallback: 0n and an empty buffer are falsy but real
projections. DOC_ORIGINALS_KEY is declared once and imported. index-key-codec's
docblock said bigints serialize to `String(value)`; corrected, and persisted
read-set keys keep tag "2" (text) exactly as on alpha — the previous commit's
number projection would have moved them to tag "1", this restores that.
serialize-sql no longer overclaims: only bigint and bytes are asserted equal,
and Date/NaN/Infinity are named as still drifting.
Tests: exact `=` above 2^53 across three adjacent values, two distinct large
bigints in a unique index, and a real .paginate() continueCursor round-trip —
7 of the file's 35 cases fail against the previous commit. The gap that let this
through was the same shape as round one: storage was tested above
MAX_SAFE_INTEGER and the query twin never was.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
There was a problem hiding this comment.
Actionable comments posted: 7
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/do/src/shard-do.ts`:
- Around line 310-314: Update decodeAdminArgs to validate the value returned by
decodeWire before casting or returning it: reject primitives, null, and arrays,
accepting only plain non-array objects. Preserve the existing
LunoraError("BAD_REQUEST", "malformed admin RPC arguments") response for both
decode failures and invalid decoded shapes before admin handlers dispatch.
In `@packages/shard-engine/src/aggregate-tally.ts`:
- Line 191: Update the group-result serialization path around encodeWire and the
groupBy row.key read boundary to decode wire-tagged keys with decodeWire after
JSON.parse. Preserve compatibility by allowing ordinary JSON keys to pass
through unchanged, so declared bigint and byte types are restored in indexed
groupBy results.
- Around line 61-69: Update the aggregate accumulation logic in
aggregate-tally.ts so each updated sum, average numerator, and maintained
minimum/maximum is checked against MAX_SAFE_BIGINT before conversion or
persistence, not just individual bigint inputs. Reuse the existing LunoraError
BAD_REQUEST behavior and preserve exact aggregation by rejecting totals outside
the safe bigint range.
In `@packages/shard-engine/src/ctx-db.ts`:
- Around line 2484-2496: Extract the bigint scan-eligibility validation from the
aggregate() path into a shared helper. Invoke it after each aggregate-index fast
path and before every SQL reducer scan, including the groupBy() scan, so SUM,
AVG, MIN, and MAX on v.bigint() fields throw the same BAD_REQUEST unless a
matching aggregate index handles them.
In `@packages/shard-engine/src/do-sql.ts`:
- Around line 89-94: Update packages/shard-engine/src/do-sql.ts lines 89-94 to
avoid rejecting pre-existing documents solely because they contain
__originals__, while adding schema-level reservation for future documents and
migrating or preflighting affected stored rows. In do-sql.ts lines 142-152,
restore originals only when the versioned internal envelope is present; apply
the same envelope check in introspect.ts lines 903-911 before Studio row
expansion.
In `@packages/shard-engine/src/estimate-bytes.ts`:
- Around line 39-42: Update the byte-handling branch in the estimator around
encodeDocJson’s contract so top-level ArrayBuffer and ArrayBuffer view values
include both the Base64 projection and the preserved wire value under
__originals__. Reuse the same document-encoding representation or sizing logic
for both stored forms, rather than charging only the Base64 payload, while
preserving the existing return behavior.
In `@packages/shard-engine/src/sql-projection.ts`:
- Around line 106-110: Replace the toBase64 usage in sqlComparableProjection()
for ArrayBuffer and ArrayBuffer.isView values with the project’s
byte-order-preserving binary encoder/serializer. Keep the existing handling of
byte offsets and lengths, and ensure indexed byte columns retain lexicographic
byte order.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 7775a822-89ab-4a96-82fb-644d253674a0
⛔ Files ignored due to path filters (2)
packages/do/__tests__/shard-do.admin.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/shard-engine/__tests__/ctx-db.bigint-bytes.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (10)
packages/do/src/shard-do.tspackages/shard-engine/src/aggregate-tally.tspackages/shard-engine/src/ctx-db.tspackages/shard-engine/src/do-sql.tspackages/shard-engine/src/estimate-bytes.tspackages/shard-engine/src/index-key-codec.tspackages/shard-engine/src/introspect.tspackages/shard-engine/src/query-args.tspackages/shard-engine/src/serialize-sql.tspackages/shard-engine/src/sql-projection.ts
| if (typeof value === "bigint") { | ||
| if (value > MAX_SAFE_BIGINT || value < -MAX_SAFE_BIGINT) { | ||
| throw new LunoraError( | ||
| "BAD_REQUEST", | ||
| `bigint ${value.toString()} exceeds Number.MAX_SAFE_INTEGER and cannot be aggregated exactly — aggregate a narrower column, or read the rows and reduce them in the handler`, | ||
| ); | ||
| } | ||
|
|
||
| return Number(value); |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win
Validate the accumulated aggregate value.
This check validates each bigint input but not the running total. 9007199254740991n and 2n both pass, but their sum cannot be represented exactly as a JavaScript number. The companion REAL value can silently round.
Validate each updated sum, average numerator, and maintained extreme before persisting it.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/shard-engine/src/aggregate-tally.ts` around lines 61 - 69, Update
the aggregate accumulation logic in aggregate-tally.ts so each updated sum,
average numerator, and maintained minimum/maximum is checked against
MAX_SAFE_BIGINT before conversion or persistence, not just individual bigint
inputs. Reuse the existing LunoraError BAD_REQUEST behavior and preserve exact
aggregation by rejecting totals outside the safe bigint range.
| if (Object.hasOwn(document, DOC_ORIGINALS_KEY)) { | ||
| throw new LunoraError( | ||
| "BAD_REQUEST", | ||
| `"${DOC_ORIGINALS_KEY}" is reserved by the row store's document encoding and cannot be used as a field name — rename the field`, | ||
| ); | ||
| } |
There was a problem hiding this comment.
🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift
Use an unambiguous metadata format for projected originals.
__originals__ was valid user data before this change. The new format cannot distinguish a legacy user object from internal metadata.
packages/shard-engine/src/do-sql.ts#L89-L94: do not make pre-existing documents unwritable solely because they use this field name.packages/shard-engine/src/do-sql.ts#L142-L152: restore originals only when a versioned internal envelope is present.packages/shard-engine/src/introspect.ts#L903-L911: apply the same envelope check before expanding rows for Studio.
Add schema-level reservation for future documents and migrate or preflight affected stored rows.
📍 Affects 2 files
packages/shard-engine/src/do-sql.ts#L89-L94(this comment)packages/shard-engine/src/do-sql.ts#L142-L152packages/shard-engine/src/introspect.ts#L903-L911
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/shard-engine/src/do-sql.ts` around lines 89 - 94, Update
packages/shard-engine/src/do-sql.ts lines 89-94 to avoid rejecting pre-existing
documents solely because they contain __originals__, while adding schema-level
reservation for future documents and migrating or preflighting affected stored
rows. In do-sql.ts lines 142-152, restore originals only when the versioned
internal envelope is present; apply the same envelope check in introspect.ts
lines 903-911 before Studio row expansion.
| if (value instanceof ArrayBuffer) { | ||
| return toBase64(new Uint8Array(value)); | ||
| } | ||
|
|
||
| return ArrayBuffer.isView(value) ? toBase64(new Uint8Array(value.buffer, value.byteOffset, value.byteLength)) : undefined; |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
🧩 Analysis chain
🏁 Script executed:
#!/bin/bash
set -euo pipefail
fd -a '^base64\.(ts|tsx|js|jsx)$' shared | while IFS= read -r file; do
echo "== $file =="
sed -n '1,240p' "$file"
done
rg -n -C 3 'sqlComparableProjection|toBase64|v\.bytes|orderBy|withIndex' packages/shard-engine sharedRepository: anolilab/lunora
Length of output: 50371
🏁 Script executed:
#!/bin/bash
set -euo pipefail
echo "== sql-projection outline =="
ast-grep outline packages/shard-engine/src/sql-projection.ts || true
echo "== sql-projection relevant lines =="
sed -n '1,180p' packages/shard-engine/src/sql-projection.ts
echo "== query-args toBase64 =="
sed -n '1,180p' packages/shard-engine/src/query-args.ts
echo "== bytes index usage =="
python3 - <<'PY'
from pathlib import Path
matches = []
for p in Path("packages/shard-engine/src").rglob("*.ts"):
text = p.read_text()
if "sqlComparableProjection" in text or "toBase64" in text:
matches.append(p)
for m in matches:
print(m)
PY
echo "== small behavior probe for btoa/std base64 order for relevant byte patterns =="
node - <<'JS'
function toBase64(bytes) {
let binary = "";
const chunk = 0x8000;
for (let index = 0; index < bytes.length; index += chunk) binary += String.fromCharCode(...bytes.subarray(index, index + chunk));
return btoa(binary);
}
const cases = [
[0xd0],
[0x00],
[0x0d, 0x00],
[0xd0, 0x00],
[0x0d, 0x01],
[0xd0, 0x01],
];
for (const bytes of cases) console.log(JSON.stringify(Object.values(bytes).map(b => b.toString(16).padStart(2,'0').toUpperCase())), "=>", toBase64(new Uint8Array(bytes)));
console.log("0xd0 < 0x00 byte order:", toBase64(new Uint8Array([0xd0])) < toBase64(new Uint8Array([0x00])));
console.log("0xd0 encodes to same prefix plus padded suffix as 0x0d when length 1:", toBase64(new Uint8Array([0xd0])).startsWith("Dc="), toBase64(new Uint8Array([0x0d])).slice(0,3));
JSRepository: anolilab/lunora
Length of output: 13618
Do not use Base64 for byte order preservation.
sqlComparableProjection() stores v.bytes() values with shared/base64.toBase64(), whose standard Base64 alphabet does not preserve byte order; for example 0xd0 encodes to 0A==, which sorts before 0x00 encoded as AA==. Use a byte-preserving binary encoder/serializer instead of Base64 for indexed byte columns.
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/shard-engine/src/sql-projection.ts` around lines 106 - 110, Replace
the toBase64 usage in sqlComparableProjection() for ArrayBuffer and
ArrayBuffer.isView values with the project’s byte-order-preserving binary
encoder/serializer. Keep the existing handling of byte offsets and lengths, and
ensure indexed byte columns retain lexicographic byte order.
Rows written between ab0afaf and the SQL-comparable projection stored v.bigint()/v.bytes() as a wire-tagged array left in place at `$.field`. They read back correctly, but the stored text is an array, so `json_extract` never matches: filter/withIndex miss them, ORDER BY sorts them as text, SUM counts them as 0. The projection fix heals a row on any patch/replace and migrates nothing on its own, so what was left is the rows nobody rewrites — for `paymentSessions`, precisely the settled sessions, which are the ones most likely to be queried by amount and least likely to be written again. No runner here, deliberately: `runDataMigration` already provides keyset batching through the normal writer, resumability, idempotence, dryRun, and the property this actually needs — cursor stability, since `replace` preserves `_id`/`_creationTime` so a rewritten row never moves relative to the cursor. This adds only what is specific to the defect: - Affected-table derivation from the schema. A table with no bigint/bytes column cannot hold an affected row and is never scanned; `.global()` tables are excluded outright, their rows live in the sql-store backend. - A detector over the STORED TEXT, because a decoded document cannot say which format it came from — that is the whole defect. The signal is the wire sentinel at element 0 of a top-level projected field, so `json_extract` returns NULL for a current row and the sentinel for a legacy one. Narrow on purpose: a predicate matching everything would rewrite every shard on every run and would look like it was working. Paths are bound, not interpolated. - An IDENTITY transform. Returning the document unchanged makes the runner rewrite it through `replace`, and `encodeDocJson` re-projects on the way down. Nothing reconstructs a value, so there is no third implementation of the projection to drift — which is what produced the original defect. A raw SQL UPDATE over `__doc__` would be faster and would bypass triggers, CDC and subscriber notification. - Rows already in the current projection return `undefined`, so the runner counts them processed and leaves them alone. Rewriting them is wasted work and pokes every subscriber on the row. The sentinel is derived from `encodeWire(0n)[0]` rather than copied: the codec does not export it and must not be edited, and a hand-copied literal would be a second definition free to drift. Invocation needs no CLI change — `lunora migrate up/status <id>` already takes an id, fans out per shard, and has --dry-run/--batch-size/--steps. Codegen resolves the reserved `__lunora_reproject__<table>` id when the app's own registry misses, built per-invocation because the detector needs the shard's `sql` handle. `ensureMigrated()` moved above the lookup so the tables exist first. Tests cover the plan's list plus the contract: a seeded legacy row becomes queryable and stores the current projection, a current row is not rewritten, a mixed table converges, a resumed run does not double-count, dryRun changes nothing, v.bytes() alongside v.bigint(), a nested tagged value is NOT matched, and a decoded document with Date/Map/Set/NaN/Infinity/array-undefined/nested bigint is byte-identical across the rewrite. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
There was a problem hiding this comment.
Actionable comments posted: 2
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/payment/docs/index.mdx`:
- Around line 388-392: Update the migration instructions near the
`__lunora_reproject__<table>` identifier to limit repetition to affected
non-global tables. Explicitly exclude tables declared with `.global()`, while
retaining the existing bigint/bytes column criteria.
In `@packages/shard-engine/src/reprojection-backfill.ts`:
- Around line 91-95: Update legacyRowPredicate so each field name is escaped
according to SQLite JSON-path rules before constructing the bound path
parameter. Preserve the existing $.<field>[0] shape and matching behavior while
ensuring names containing dots or other path-significant characters are treated
as a single top-level key.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 3b2efc29-2312-4852-a5ad-ff829161aeb0
⛔ Files ignored due to path filters (18)
api-snapshots/do.api.mdis excluded by none and included by noneapi-snapshots/lunora.api.mdis excluded by none and included by noneapi-snapshots/shard-engine.api.mdis excluded by none and included by noneexamples/auth-playground/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by noneexamples/blog/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by noneexamples/chess/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by noneexamples/expo/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by noneexamples/feedback-board/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by noneexamples/kanban-board/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by noneexamples/notify-demo/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by noneexamples/offline-rejections/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by noneexamples/payment-demo/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by noneexamples/realtime-cursors/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by noneexamples/tanstack-start/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by noneexamples/team-chat/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by noneexamples/todo-app/lunora/_generated/shard.tsis excluded by!**/_generated/**and included by nonepackages/codegen/__tests__/fixtures/simple/expected/_generated/shard.tsis excluded by!**/_generated/**,!**/__tests__/**and included bypackages/**packages/shard-engine/__tests__/reprojection-backfill.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (5)
packages/codegen/src/emit.tspackages/do/src/index.tspackages/payment/docs/index.mdxpackages/shard-engine/src/index.tspackages/shard-engine/src/reprojection-backfill.ts
Completed plans do not stay in this directory - the record is the README row and git history. 311 shipped inside #365, so its file would have been born and deleted in a day. Its one live thread, wiring countLegacyRows into doctor, moves into the row so it does not vanish with the file. 312 stays: its design decisions need a measurement first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
One root cause under four HIGH findings: `encodeDocJson` projects on the value's
runtime type (`typeof value === "bigint"`), while every guard and the whole
backfill dispatched on the declared validator `kind`. `v.optional(v.bigint())`
has `kind === "optional"` with the real validator on `_meta.inner`, so those two
disagreed, and each disagreement was a silent wrong answer:
- `aggregate` sum over an optional bigint reduced the padded keys — 100n + 250n
read as 2e+39, no throw.
- `groupBy`'s scan path had no guard at all, so a PLAIN bigint did the same, and
its group keys came back as 40 characters of zero padding.
- `reprojectableFields` returned [] for an optional bigint, so `countLegacyRows`
read 0 and `buildReprojectionMigration` returned undefined. The docs tell an
operator that a second --dry-run reading 0 means nothing is left, so an
affected app was told it was clean — a false all-clear on a data-migration
completeness check.
The repo already owned the correct predicate (`effectiveColumnKind` in
sql-store, added for this same plan). It now lives once in
`shared/effective-kind` — neither store depends on the other, and a fourth copy
is what caused this. sql-store's export becomes a thin alias.
`isProjectedKind` / `mayHoldProjectedValue` build on it: the guards use the
narrow one, the backfill the wide one, which also counts `v.any()`/`v.union()`/
`v.from()` since an untyped field can hold a bigint the projection then projects.
Widening the backfill is only safe because the row predicate now tests the tagged
value's TYPE slot rather than merely the sentinel — a `Date` under a `v.any()`
field is legitimately tagged today, and matching it would rewrite the row, re-tag
the Date, and leave it "legacy" forever.
Also fixed:
- The backfill was unreachable. `resolveMigrationTable` AST-scans the user's
`lunora/` for `defineMigration({ id })`, so a reserved framework id exited 1
with no network call. The CLI now derives the table from the id. The prefix
moved to `shared/reprojection-id` because the CLI depends on neither the engine
nor `@lunora/do`, and spelling it twice would surface as MIGRATION_NOT_FOUND
with nothing pointing at the cause.
- Studio grid regression. `adminResponse` encodes every admin result and
`expandDocumentRows` spread ALREADY-tagged originals, so the codec escaped them
and the grid rendered the tag instead of `4200n`. It decodes first now, which
fixes the POST and WS paths together. Two comments asserted the opposite of
what the code did — one still described the world before `adminResponse`
existed; both rewritten.
- One oversize legacy row wedged the backfill forever: the legacy format stored a
bigint of any width (a uint256 balance is 78 digits, and this repo ships
`@lunora/x402`), the transform threw, and the run persisted `failed` with the
cursor pinned so every resume died identically. It skips and reports now.
- Indexed `groupBy` returned the tagged array as the group key (`encodeAggregateKey`
writes `encodeWire`, the reader was a bare `JSON.parse`) — on alpha this threw,
so it was a loud failure turned into a wrong answer.
- `estimateBytes` undercounted by ~6x for bigint and ~2x for bytes (measured),
because a projected column is stored TWICE. Both copies are charged now, and
the docblock states the ~4x/~2x on-disk cost the branch never mentioned.
- A field named `__proto__` lost its value to `Object.prototype`'s setter; the
originals map is null-prototype.
- Comments corrected where they overclaimed: base64 is NOT order-preserving, so
`ORDER BY` on a `v.bytes()` column is ordered by base64 text (equality and the
backfill are unaffected); the aggregate guard bounds each contribution, not the
running total, which rounds in the REAL column exactly as a `v.number()` sum
does; binding a JSON path defeats SQL injection but not SQLite's path grammar,
so field names are quoted; 39 digits clears uint128's ~3.4e38, not 1.7e38.
From CodeRabbit, additionally: admin args are re-checked for object-ness after
decode (a tagged `undefined` or a root array changes the SHAPE, and handlers
index before validating), and the payment docs no longer send operators at
`.global()` tables, which correctly answer MIGRATION_NOT_FOUND.
Tests: an ordering property test over 211 values (~44.5k pairs) asserting
`sign(key(a) cmp key(b)) === sign(a - b)`, checked in JS and again under a real
SQLite ORDER BY; the 39-digit refusal; `v.optional(v.bigint())` end to end. Fixed
three broken ones — a "no-op on a second full run" that never ran twice, a
"groups by a bigint column" that called `count`, and four assertions comparing
the projection against itself via `serializeSqlValue`, which would have passed
had the projection reverted to `String(value)`.
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/shard-engine/src/ctx-db.ts`:
- Around line 962-965: Update assertReducibleBySql to use
mayHoldProjectedValue(validator) instead of only checking exact projected kinds,
rejecting broad validators such as unions, any, or from-based validators that
may contain runtime bigint or bytes values while preserving the existing
SQL-reducibility validation behavior.
🪄 Autofix
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 78caab4f-bcb9-4bcc-8317-a5ef3d999501
⛔ Files ignored due to path filters (5)
packages/shard-engine/__tests__/ctx-db.bigint-bytes.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/shard-engine/__tests__/reprojection-backfill.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/shard-engine/__tests__/sql-projection.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**shared/effective-kind.tsis excluded by none and included by noneshared/reprojection-id.tsis excluded by none and included by none
📒 Files selected for processing (13)
packages/cli/src/commands/migrate/handler.tspackages/cli/tsconfig.jsonpackages/do/src/shard-do.tspackages/payment/docs/index.mdxpackages/shard-engine/src/aggregate-tally.tspackages/shard-engine/src/ctx-db.tspackages/shard-engine/src/do-sql.tspackages/shard-engine/src/estimate-bytes.tspackages/shard-engine/src/index-key-codec.tspackages/shard-engine/src/introspect.tspackages/shard-engine/src/reprojection-backfill.tspackages/shard-engine/src/sql-projection.tspackages/sql-store/src/value-codec.ts
🚧 Files skipped from review as they are similar to previous changes (7)
- packages/shard-engine/src/index-key-codec.ts
- packages/shard-engine/src/aggregate-tally.ts
- packages/payment/docs/index.mdx
- packages/shard-engine/src/do-sql.ts
- packages/shard-engine/src/introspect.ts
- packages/shard-engine/src/estimate-bytes.ts
- packages/do/src/shard-do.ts
| const assertReducibleBySql = (definition: TableDefinitionLike, field: string, label: string): void => { | ||
| const validator = definition.shape[field]; | ||
|
|
||
| if (validator && isProjectedKind(validator)) { |
There was a problem hiding this comment.
🎯 Functional Correctness | 🟠 Major | ⚡ Quick win
Reject scan aggregates for validators that can hold projected values.
Lines 963-965 only reject exact bigint and bytes validators. A v.union(), v.any(), or v.from() field can contain a runtime bigint or byte value. That value is stored as a projection, but the aggregate and groupBy() scan paths then reduce or group the projection key.
Use mayHoldProjectedValue() here, or add an equivalent safe scan check. This prevents SUM, MIN, MAX, and GROUP BY from returning projection keys for broad validators.
Proposed fix
-import { isProjectedKind } from "./sql-projection";
+import { mayHoldProjectedValue } from "./sql-projection";
...
- if (validator && isProjectedKind(validator)) {
+ if (validator && mayHoldProjectedValue(validator)) {📝 Committable suggestion
‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.
| const assertReducibleBySql = (definition: TableDefinitionLike, field: string, label: string): void => { | |
| const validator = definition.shape[field]; | |
| if (validator && isProjectedKind(validator)) { | |
| const assertReducibleBySql = (definition: TableDefinitionLike, field: string, label: string): void => { | |
| const validator = definition.shape[field]; | |
| if (validator && mayHoldProjectedValue(validator)) { |
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
In `@packages/shard-engine/src/ctx-db.ts` around lines 962 - 965, Update
assertReducibleBySql to use mayHoldProjectedValue(validator) instead of only
checking exact projected kinds, rejecting broad validators such as unions, any,
or from-based validators that may contain runtime bigint or bytes values while
preserving the existing SQL-reducibility validation behavior.
Two differences from `shared/json-path-segment.ts` on the branch that fixes this repo-wide, both now removed: - `\` was not escaped, so a backslash-bearing field yielded `$."back\slash"`, which SQLite 3.53 resolves to NULL rather than erroring — the same silent miss the quoting exists to prevent. It is escaped BEFORE `"`, or the backslash the quote escape inserts gets doubled in turn. - It quoted unconditionally; it now emits the bare form for a field matching the identifier regex `parse-validator.ts` already enforces on every schema field. Neither was reachable here — the backfill's field names come from the schema shape and codegen constrains those to identifiers — so this is drift prevention, not a bug fix. Two copies of an escaping rule diverging in opposite directions is the defect the shared helper exists to fix, and one copy being unreachable today is how that survives to become reachable later. Kept local rather than imported: the shared file exists only on the other branch, and importing it would turn a soft merge-order note into a hard dependency between two otherwise independent PRs. Whichever lands second deletes this copy, and because the two now agree exactly, that deletion is a no-op refactor. `jsonPathSegment` is exported from the module for its test only, deliberately not re-exported from the barrel, so the public API surface is unchanged. Tests cover both differences plus a SQLite round-trip proving the emitted path actually resolves — all three fail against the previous implementation. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
* docs(plans): file the two follow-ups from the bigint fix 311 is the data the defect left behind: rows written between ab0afaf and #365 hold the bigint tagged in place, so they read correctly but no json_extract comparison matches them. #365 heals a row on any write and migrates nothing, which leaves exactly the rows nobody rewrites - for paymentSessions, the settled sessions. runDataMigration already gives resumability, idempotence and cursor stability, so the work is a transform and a detector, not a runner. 312 is the price #365 paid. A soft-delete table can never reach the aggregate companion, and declaring aggregateIndex on a bigint column caps that column's writes at 2^53. Both throw rather than return a wrong answer, and closing either by rounding is a STOP condition. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP * docs(plans): record 311 as implemented on the 365 branch Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP * docs(plans): sync the index with 311 shipping inside 365 Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP * docs(plans): retire 311, keep 312 Completed plans do not stay in this directory - the record is the README row and git history. 311 shipped inside #365, so its file would have been born and deleted in a day. Its one live thread, wiring countLegacyRows into doctor, moves into the row so it does not vanish with the file. 312 stays: its design decisions need a measurement first. Co-Authored-By: Claude Opus 5 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP --------- Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
Note
Ready to merge. This PR opened as a draft because a thermo-nuclear review found three HIGH findings — all three are fixed in
3c6523096:Number()is gone, and a bigint now stores as a fixed-width order-preserving decimal key. The findings and their fixes are kept below as the record. The backfill that completes the fix ships here too (bb398f461), so merging closes the data exposure rather than opening a follow-up. Two tracked consequences are at the bottom.The defect being fixed
ab0afaf00(#338) madev.bigint()andv.bytes()storable on the Durable Object row store by routing the__doc__blob through the wire codec. That fixed the throw and the silent{}, and introduced a worse defect: it madev.bigint()unqueryable.Storage held the tagged array
["$lunora.wire$","bigint","10"]while every query path bound its comparison value throughserializeSqlValue, which rendered"10".json_extractnever matched. Executed on anode:sqliteharness againstalpha:alphafilter { amount: 10n }[]filter { amount: { gt: 9n } }orderBy amount asc10, 200, 9(lexical)aggregate sumnull/0aggregate max'["$lunora.wire$","bigint","9"]'leaked to the callercount0A balance check reads a confident, wrong zero.
@lunora/payment'spaymentSessionsdeclares threev.bigint()money columns on a shard-local table.The approach
Teaching
serializeSqlValuethe tagged form is a six-line diff and fixes equality only — a tagged array is still text to SQLite, so ranges,ORDER BY,SUMandMIN/MAXstay wrong and stay silent. InsteadencodeDocJsonprojects top-level bigint/bytes to a SQL-comparable scalar and parks the wire-tagged originals under a reserved key, andserializeSqlValuebinds the same projection.json_extractthen returns an INTEGER and every operation above is numeric.Also in scope: CDC egress re-encodes (
cdcSync→encodeWire,applyCdc→decodeWire), andestimateBytessizes bigint/bytes instead of refusing them — with plan 270 landed, refusing them made a bigint insert fail withBAD_REQUESToutright.Two findings from the original review proved not live and were deliberately not "fixed":
sql-store'ssqliteDecodealready has its bytes branch (and copies through an ownedUint8Array, which is more correct than the plan's.slice()— that preserves species and hands back aSharedArrayBufferfor shared-memory views, whichv.bytes()rejects), and theintrospect.tsitem is a documented decision from plan 300.The encoding, and why the obvious candidates lose
The first attempt projected a bigint through
Number(). That fixed the silent-empty-result bug by introducing a worse one: above 2^53 distinct values collapse, soeqmatched rows that were not equal (false positives, wherealphawas uniformly wrong in the safe direction), auniqueindex rejected two distinct bigints as a duplicate, and.paginate()threw a rawTypeErrorfrom a bareJSON.stringifyin the cursor encoder.Three candidates were tested against a real SQLite build:
json_extract— but reading such a row back throwsRangeError: Value is too large to be represented as a JavaScript numberout of the driver, so the row becomes unreadable. The comparison side still cannot bind an exact int64.=, but sorts"9"after"10".=exact at 2^53 and either side,ORDER BYnumeric across a 22-digit range including negatives, ranges narrow correctly. 39 digits covers the whole unsigned 128-bit range; past it the write is refused rather than silently mis-sorted.The decision rule was: never return a non-matching row; exactness beats operation coverage; if an operation cannot be exact, fail loudly rather than degrade silently.
The cost is
SUMon the scan path. Padded text is not something SQLite can add up, so the aggregate reader refuses av.bigint()field on the scan path rather than return the 1.5e40 that falls out of coercing it, and the companion refuses a value its REAL column cannot hold exactly.aggregateIndexis the supported way to sum a bigint column.Two consequences to weigh
aggScopedisables the indexed fast path unconditionally, so every aggregate on such a table hits the scan. With the new guard that means no aggregate over av.bigint()column on a soft-delete table, at any magnitude. Not a regression (the scan answer was garbage either way), but a real hole, and it is not closed here.aggregateIndexon a bigint column now refuses to store values past 2^53 — the write fails, not just the read. Correct under the rule above, but one declaration makes the column unusable for large values.Other findings
exportShardis the un-fixed twin of thecdcSyncfix in this very commit: it returns decoded rows throughjsonResponse, so a bigint column 500s a shard export and a bytes column reaches a backup as{}.importShardlikewise lacks thedecodeWirethatapplyCdcgained. Pre-existing, but this commit's own rationale applies verbatim.aggregate sum/maxaboveMAX_SAFE_INTEGERnow throw an unmappedRangeError(was a silent0). Directionally right, but unmapped and untested, andavgis inconsistent.__sql__guard is prospective only, yet the docblock claims a historical invariant ("can never have been stored"). A pre-existing row with that field is silently restructured on read and the damage persisted on first patch.alphashowed an obviously-broken tagged array.All fixed.
exportShardwas fixed at the boundary rather than as a second special case — oneadminResponseencode covers every admin handler and onedecodeAdminArgscovers every ingress, so the 12-line inline comment is gone. The projection is now one function (sql-projection.ts).__sql__→__originals__, its guard documented as prospective, and decode claims the key only for a plain object so a legacy scalar is left alone rather than exploded.index-key-codecis corrected — and note persisted read-set keys keep tag"2"(text), exactly as onalpha: the previous commit's number projection would have moved them to tag"1", and this restores it.Verified clean
OCC/CAS byte-identity holds across 19 value shapes including a literal
__proto__key and mixed key order; companion formats (__sort_k<i>__,__key__) self-heal via TRUNCATE+rebuild on first write, so no rollout migration; the__sql__guard is not bypassable for new writes (insert, patch,Object.create(null),JSON.parseown property); bytes edge cases pass including non-zerobyteOffsetandSharedArrayBuffer-backed views.shard-engine1002,do523,sql-store105,payment180,d1243,runtime846,observability224 — all green.api:check47 snapshots.The backfill ships here too (plan 311)
Rows written between
ab0afaf00and this branch hold the bigint tagged in place: they read back correctly, but$.fieldis still array text, sofilter/withIndex/ORDER BY/SUMmiss them.patch/replaceheal a row, so the exposure is the rows nobody rewrites — forpaymentSessions, the settled sessions that are never touched again and are exactly what an amount query looks for.packages/shard-engine/src/reprojection-backfill.ts(~175 lines) closes that, and no batch runner was written:runDataMigrationalready owns batching, resumption, the cursor, claim/heartbeat anddryRun. The new module contributes three things.bigint/bytesfield.json_extract(__doc__, '$.f[0]')returns the wire sentinel for a legacy array andNULLfor a current row, whose field is a JSON string. Paths are bound, not interpolated, so a field name needs no escaping rule of its own, andWIRE_TAGis derived fromencodeWire(0n)[0]rather than hand-copied — a literal here would be a second definition free to drift.replaceit andencodeDocJsonre-projects on the way down, so nothing reconstructs a value and there is no third implementation of the projection to drift. Rows already current returnundefined, so they are counted and left alone rather than pointlessly poking every subscriber.A test pins the STOP condition directly: a row whose only tagged value is nested is not matched. Nested values stay wire-tagged under the current projection, so a detector that scanned the blob for the sentinel would rewrite every table forever while looking like it worked.
Invocation needed no CLI change —
lunora migrate up|status <id>already fans out per shard with--dry-run/--batch-size; codegen resolves the reserved__lunora_reproject__<table>when the app's own registry misses.Two answers for reviewers
countLegacyRowsis one query per affected table and reads exactly zero when a table is fully re-projected — the same figure--dry-runreports before a run and the one an operator wants after. It plusreprojectionTablesare exported forlunora doctor, but doctor is not wired: it was outside the plan's workstreams and needs a per-shard fan-out decision.Known cost
Detection is one primary-key seek per visited row, riding alongside a scan that already reads every row. The alternative — pre-computing the legacy id set in one query — holds every affected id in a Durable Object's 128 MB. Marked with a
ponytail:comment naming the ceiling.Second review round (
59b09aed8)A full-diff thermo pass plus CodeRabbit found a fourth defect class, with one root cause:
encodeDocJsonprojects on the value's runtime type, while every guard and the whole backfill dispatched on the declared validatorkind. Wherever those disagreed the result was a silent wrong answer —v.optional(v.bigint())reduced to2e+39instead of350, with no throw, andgroupBy's scan path had no guard at all.Fixed once:
effectiveColumnKind(which already existed insql-store, added for this same plan) now lives inshared/effective-kind, with sql-store's export as a thin alias, and two predicates build on it —isProjectedKindfor the guards,mayHoldProjectedValuefor the backfill. A fourth copy of that predicate is what caused this.groupByassertReducibleBySqlat all three SQL-reducing entry pointsshared/reprojection-id)v.any()/v.union()/v.from()alpha)expandDocumentRowsdecodes beforeadminResponse, fixing POST and WS togetherWidening the backfill to
v.any()was only safe alongside tightening the row predicate to test the tagged value's type slot rather than just the sentinel — otherwise aDateunder av.any()field would match, be rewritten, be re-tagged, and stay "legacy" forever.MEDIUM/LOW: oversize legacy rows now skip-and-report instead of pinning the cursor and wedging the table forever (a uint256 balance is 78 digits and this repo ships
@lunora/x402); bothgroupBykey paths fixed; the base64 ordering claim corrected;estimateBytesmeasured at ~6–9.6× under for bigint and ~2× for bytes, and fixed;__proto__handled with a null-prototype map. M4 was corrected rather than fixed: the companion's running total rounds exactly as av.number()sum does — that is the companion's precision model, not something bigint-specific — so the docblock now says so instead of implying a guarantee.CodeRabbit contributed four findings neither reviewer had; all fixed. It also surfaced that
documentPathhas the same JSON-path quoting hole, so a field nameda.bis already unaddressable everywhere in this store — pre-existing, changes everyCREATE INDEXexpression, and needs its own rebuild story. Flagged, not fixed.Why
__originals__was keptBoth projections are reversible, so a discriminator inside the value looks like it would delete the reserved key, its guard, the
decodeDocJsonheuristic, theintrospectspecial case and the storage doubling. It was rejected: the discriminator has to live at$.field, which is the thing SQL compares, soserializeSqlValuemust emit it too and the ordering property must be re-proved across the marker boundary — and any marker sorting between two magnitudes reintroduces exactly the mis-ordering this branch exists to remove. It also has to be unforgeable against av.any()field holding a string that starts with it. Worth its own plan and its own property test; not a fold-in.This branch added a local
jsonPathSegmentinreprojection-backfill.ts:56to quote JSON-path segments in the backfill's detector. #371 makes that a shared helper (shared/json-path-segment.ts) and fixes two things this local copy gets wrong: it does not escape\, and it quotes unconditionally where the shared one quotes only when the bare form would misparse.Neither is reachable from here — the backfill's field names come from the schema shape, and codegen (
parse-validator.ts:247) already guarantees those are identifier-safe. But two copies drifting in opposite directions is exactly the defect #371 exists to fix.Whichever of the two merges second should delete the local copy and import the shared one. Stated here rather than tracked out of band, because the last merge-order dependency in this series was known and still fired on
alpha.Verification
shard-engine 1019 (+12) · do 523 · codegen 1068 · payment 180 · sql-store 105 · d1 243 · runtime 846 · observability 224.
lint:eslint --max-warnings=0+lint:typeson shard-engine/do/codegen.api:check✅ 47 after a fresh build (three snapshots gained only the five new exports).lint:package-json✅ 74. Codegen fixture and all 13 example_generated/shard.tsregenerated.Unrelated pre-existing drift found and deliberately not folded in:
examples/payment-demo'sdataModel.tsanddrizzle.shard.tsare stale onalpha(@lunora/paymentgained ausageEvents.modefield that was never regenerated). Reverted here to keep the diff honest; worth a separate cleanup.🤖 Generated with Claude Code
https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
Summary by CodeRabbit
New Features
bigintand binary values across administration, storage, queries, cursors, imports, exports, and introspection.bigintand binary data.Bug Fixes
Documentation