Skip to content

fix(shard-engine): make v.bigint() columns queryable, not just storable - #365

Merged
prisis merged 9 commits into
alphafrom
fix/265-wire-codec-query-parity
Aug 8, 2026
Merged

prisis merged 9 commits into
alphafrom
fix/265-wire-codec-query-parity

Conversation

@prisis

@prisis prisis commented Aug 8, 2026 •

Copy link
Copy Markdown
Member

Note

Ready to merge. This PR opened as a draft because a thermo-nuclear review found three HIGH findings — all three are fixed in 3c6523096: Number() is gone, and a bigint now stores as a fixed-width order-preserving decimal key. The findings and their fixes are kept below as the record. The backfill that completes the fix ships here too (bb398f461), so merging closes the data exposure rather than opening a follow-up. Two tracked consequences are at the bottom.

The defect being fixed

ab0afaf00 (#338) made v.bigint() and v.bytes() storable on the Durable Object row store by routing the __doc__ blob through the wire codec. That fixed the throw and the silent {}, and introduced a worse defect: it made v.bigint() unqueryable.

Storage held the tagged array ["$lunora.wire$","bigint","10"] while every query path bound its comparison value through serializeSqlValue, which rendered "10". json_extract never matched. Executed on a node:sqlite harness against alpha:

operation result on alpha
filter { amount: 10n } []
filter { amount: { gt: 9n } } all rows
orderBy amount asc 10, 200, 9 (lexical)
aggregate sum null / 0
aggregate max the raw string '["$lunora.wire$","bigint","9"]' leaked to the caller
count 0

A balance check reads a confident, wrong zero. @lunora/payment's paymentSessions declares three v.bigint() money columns on a shard-local table.

The approach

Teaching serializeSqlValue the tagged form is a six-line diff and fixes equality only — a tagged array is still text to SQLite, so ranges, ORDER BY, SUM and MIN/MAX stay wrong and stay silent. Instead encodeDocJson projects top-level bigint/bytes to a SQL-comparable scalar and parks the wire-tagged originals under a reserved key, and serializeSqlValue binds the same projection. json_extract then returns an INTEGER and every operation above is numeric.

Also in scope: CDC egress re-encodes (cdcSync → encodeWire, applyCdc → decodeWire), and estimateBytes sizes bigint/bytes instead of refusing them — with plan 270 landed, refusing them made a bigint insert fail with BAD_REQUEST outright.

Two findings from the original review proved not live and were deliberately not "fixed": sql-store's sqliteDecode already has its bytes branch (and copies through an owned Uint8Array, which is more correct than the plan's .slice() — that preserves species and hands back a SharedArrayBuffer for shared-memory views, which v.bytes() rejects), and the introspect.ts item is a documented decision from plan 300.

The encoding, and why the obvious candidates lose

The first attempt projected a bigint through Number(). That fixed the silent-empty-result bug by introducing a worse one: above 2^53 distinct values collapse, so eq matched rows that were not equal (false positives, where alpha was uniformly wrong in the safe direction), a unique index rejected two distinct bigints as a duplicate, and .paginate() threw a raw TypeError from a bare JSON.stringify in the cursor encoder.

Three candidates were tested against a real SQLite build:

Candidate Verdict
Raw JSON integer literal SQLite really does hold exact int64 through json_extract — but reading such a row back throws RangeError: Value is too large to be represented as a JavaScript number out of the driver, so the row becomes unreadable. The comparison side still cannot bind an exact int64.
Plain decimal text Exact for =, but sorts "9" after "10".
Fixed-width order-preserving key ✅ Sign character, magnitude zero-padded to 39 digits, nines-complemented under a lower sign character when negative. = exact at 2^53 and either side, ORDER BY numeric across a 22-digit range including negatives, ranges narrow correctly. 39 digits covers the whole unsigned 128-bit range; past it the write is refused rather than silently mis-sorted.

The decision rule was: never return a non-matching row; exactness beats operation coverage; if an operation cannot be exact, fail loudly rather than degrade silently.

The cost is SUM on the scan path. Padded text is not something SQLite can add up, so the aggregate reader refuses a v.bigint() field on the scan path rather than return the 1.5e40 that falls out of coercing it, and the companion refuses a value its REAL column cannot hold exactly. aggregateIndex is the supported way to sum a bigint column.

Two consequences to weigh

  • A soft-delete table cannot use the aggregate companion at all — aggScope disables the indexed fast path unconditionally, so every aggregate on such a table hits the scan. With the new guard that means no aggregate over a v.bigint() column on a soft-delete table, at any magnitude. Not a regression (the scan answer was garbage either way), but a real hole, and it is not closed here.
  • A table declaring aggregateIndex on a bigint column now refuses to store values past 2^53 — the write fails, not just the read. Correct under the rule above, but one declaration makes the column unusable for large values.

Other findings

  • MEDIUM — exportShard is the un-fixed twin of the cdcSync fix in this very commit: it returns decoded rows through jsonResponse, so a bigint column 500s a shard export and a bytes column reaches a backup as {}. importShard likewise lacks the decodeWire that applyCdc gained. Pre-existing, but this commit's own rationale applies verbatim.
  • MEDIUM — aggregate sum/max above MAX_SAFE_INTEGER now throw an unmapped RangeError (was a silent 0). Directionally right, but unmapped and untested, and avg is inconsistent.
  • MEDIUM — the __sql__ guard is prospective only, yet the docblock claims a historical invariant ("can never have been stored"). A pre-existing row with that field is silently restructured on read and the damage persisted on first patch.
  • MEDIUM — the Studio data browser now shows a plausible-but-wrong integer where alpha showed an obviously-broken tagged array.
    All fixed. exportShard was fixed at the boundary rather than as a second special case — one adminResponse encode covers every admin handler and one decodeAdminArgs covers every ingress, so the 12-line inline comment is gone. The projection is now one function (sql-projection.ts). __sql__ → __originals__, its guard documented as prospective, and decode claims the key only for a plain object so a legacy scalar is left alone rather than exploded. index-key-codec is corrected — and note persisted read-set keys keep tag "2" (text), exactly as on alpha: the previous commit's number projection would have moved them to tag "1", and this restores it.

Verified clean

OCC/CAS byte-identity holds across 19 value shapes including a literal __proto__ key and mixed key order; companion formats (__sort_k<i>__, __key__) self-heal via TRUNCATE+rebuild on first write, so no rollout migration; the __sql__ guard is not bypassable for new writes (insert, patch, Object.create(null), JSON.parse own property); bytes edge cases pass including non-zero byteOffset and SharedArrayBuffer-backed views. shard-engine 1002, do 523, sql-store 105, payment 180, d1 243, runtime 846, observability 224 — all green. api:check 47 snapshots.

The backfill ships here too (plan 311)

Rows written between ab0afaf00 and this branch hold the bigint tagged in place: they read back correctly, but $.field is still array text, so filter/withIndex/ORDER BY/SUM miss them. patch/replace heal a row, so the exposure is the rows nobody rewrites — for paymentSessions, the settled sessions that are never touched again and are exactly what an amount query looks for.

packages/shard-engine/src/reprojection-backfill.ts (~175 lines) closes that, and no batch runner was written: runDataMigration already owns batching, resumption, the cursor, claim/heartbeat and dryRun. The new module contributes three things.

  • Affected-table derivation — only tables whose shape declares a bigint/bytes field.
  • A detector on the stored text, because a decoded document cannot say which format it came from. json_extract(__doc__, '$.f[0]') returns the wire sentinel for a legacy array and NULL for a current row, whose field is a JSON string. Paths are bound, not interpolated, so a field name needs no escaping rule of its own, and WIRE_TAG is derived from encodeWire(0n)[0] rather than hand-copied — a literal here would be a second definition free to drift.
  • An identity transform. Returning the document unchanged makes the runner replace it and encodeDocJson re-projects on the way down, so nothing reconstructs a value and there is no third implementation of the projection to drift. Rows already current return undefined, so they are counted and left alone rather than pointlessly poking every subscriber.

A test pins the STOP condition directly: a row whose only tagged value is nested is not matched. Nested values stay wire-tagged under the current projection, so a detector that scanned the blob for the sentinel would rewrite every table forever while looking like it worked.

Invocation needed no CLI change — lunora migrate up|status <id> already fans out per shard with --dry-run/--batch-size; codegen resolves the reserved __lunora_reproject__<table> when the app's own registry misses.

Two answers for reviewers

  • Not automatic on deploy. Every rewrite goes through the writer, so it fires triggers, appends CDC and pokes every subscriber on the row — on an all-legacy shard that is a full-table thundering herd, at deploy time, on a path the operator did not ask for. The "nobody ran it" failure is real but detectable and recoverable; a surprise rewrite is neither. Opt-in per app with a batch cap would change that, but it is a different design, not a flag.
  • A completeness check exists and is cheap. countLegacyRows is one query per affected table and reads exactly zero when a table is fully re-projected — the same figure --dry-run reports before a run and the one an operator wants after. It plus reprojectionTables are exported for lunora doctor, but doctor is not wired: it was outside the plan's workstreams and needs a per-shard fan-out decision.

Known cost

Detection is one primary-key seek per visited row, riding alongside a scan that already reads every row. The alternative — pre-computing the legacy id set in one query — holds every affected id in a Durable Object's 128 MB. Marked with a ponytail: comment naming the ceiling.

Second review round (59b09aed8)

A full-diff thermo pass plus CodeRabbit found a fourth defect class, with one root cause: encodeDocJson projects on the value's runtime type, while every guard and the whole backfill dispatched on the declared validator kind. Wherever those disagreed the result was a silent wrong answer — v.optional(v.bigint()) reduced to 2e+39 instead of 350, with no throw, and groupBy's scan path had no guard at all.

Fixed once: effectiveColumnKind (which already existed in sql-store, added for this same plan) now lives in shared/effective-kind, with sql-store's export as a thin alias, and two predicates build on it — isProjectedKind for the guards, mayHoldProjectedValue for the backfill. A fourth copy of that predicate is what caused this.

Finding Fix
H1 Silent wrong sums via optional kind + unguarded groupBy assertReducibleBySql at all three SQL-reducing entry points
H2 Backfill unreachable — the documented CLI command exited 1 with no network call CLI derives the table from the reserved id (shared/reprojection-id)
H3 Completeness check inverted — an affected app was told it was clean Falls out of the shared predicate; backfill now also scans v.any()/v.union()/v.from()
H4 Studio grid double-tagged bigint cells (a regression vs alpha) expandDocumentRows decodes before adminResponse, fixing POST and WS together

Widening the backfill to v.any() was only safe alongside tightening the row predicate to test the tagged value's type slot rather than just the sentinel — otherwise a Date under a v.any() field would match, be rewritten, be re-tagged, and stay "legacy" forever.

MEDIUM/LOW: oversize legacy rows now skip-and-report instead of pinning the cursor and wedging the table forever (a uint256 balance is 78 digits and this repo ships @lunora/x402); both groupBy key paths fixed; the base64 ordering claim corrected; estimateBytes measured at ~6–9.6× under for bigint and ~2× for bytes, and fixed; __proto__ handled with a null-prototype map. M4 was corrected rather than fixed: the companion's running total rounds exactly as a v.number() sum does — that is the companion's precision model, not something bigint-specific — so the docblock now says so instead of implying a guarantee.

CodeRabbit contributed four findings neither reviewer had; all fixed. It also surfaced that documentPath has the same JSON-path quoting hole, so a field named a.b is already unaddressable everywhere in this store — pre-existing, changes every CREATE INDEX expression, and needs its own rebuild story. Flagged, not fixed.

Why __originals__ was kept

Both projections are reversible, so a discriminator inside the value looks like it would delete the reserved key, its guard, the decodeDocJson heuristic, the introspect special case and the storage doubling. It was rejected: the discriminator has to live at $.field, which is the thing SQL compares, so serializeSqlValue must emit it too and the ordering property must be re-proved across the marker boundary — and any marker sorting between two magnitudes reintroduces exactly the mis-ordering this branch exists to remove. It also has to be unforgeable against a v.any() field holding a string that starts with it. Worth its own plan and its own property test; not a fold-in.

⚠️ Merge-order dependency with #371

This branch added a local jsonPathSegment in reprojection-backfill.ts:56 to quote JSON-path segments in the backfill's detector. #371 makes that a shared helper (shared/json-path-segment.ts) and fixes two things this local copy gets wrong: it does not escape \, and it quotes unconditionally where the shared one quotes only when the bare form would misparse.

Neither is reachable from here — the backfill's field names come from the schema shape, and codegen (parse-validator.ts:247) already guarantees those are identifier-safe. But two copies drifting in opposite directions is exactly the defect #371 exists to fix.

Whichever of the two merges second should delete the local copy and import the shared one. Stated here rather than tracked out of band, because the last merge-order dependency in this series was known and still fired on alpha.

Verification

shard-engine 1019 (+12) · do 523 · codegen 1068 · payment 180 · sql-store 105 · d1 243 · runtime 846 · observability 224. lint:eslint --max-warnings=0 + lint:types on shard-engine/do/codegen. api:check ✅ 47 after a fresh build (three snapshots gained only the five new exports). lint:package-json ✅ 74. Codegen fixture and all 13 example _generated/shard.ts regenerated.

Unrelated pre-existing drift found and deliberately not folded in: examples/payment-demo's dataModel.ts and drizzle.shard.ts are stale on alpha (@lunora/payment gained a usageEvents.mode field that was never regenerated). Reverted here to keep the diff honest; worth a separate cleanup.

🤖 Generated with Claude Code

https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP

Summary by CodeRabbit

  • New Features

    • Improved support for bigint and binary values across administration, storage, queries, cursors, imports, exports, and introspection.
    • Added stable ordering and comparison for large numeric and binary values.
    • Added tools to detect, count, and safely backfill legacy bigint and binary data.
  • Bug Fixes

    • Improved byte-size estimation and preservation of non-JSON values.
    • Added clearer validation for malformed, unsupported, cyclic, or oversized values.
    • Prevented unsupported aggregate operations from producing incorrect results.
  • Documentation

    • Added guidance for identifying and repairing affected historical data.

prisis and others added 2 commits August 8, 2026 14:16
The doc-blob codec stored a `v.bigint()` as the wire codec's tagged ARRAY
(`["$lunora.wire$","bigint","10"]`), but every read of a field goes through
`json_extract(__doc__, '$.field')`, and the query side binds its comparison
value through `serializeSqlValue`, which rendered the decimal string "10".
The two could never match. Measured against a real SQLite engine, with three
rows of 9/10/200:

    filter eq 10n      -> []              (expected ["s10"])
    filter gt 9n       -> all three rows  (expected 10 and 200)
    order by asc       -> 10, 200, 9
    aggregate sum      -> null / 0
    aggregate max      -> the raw string '["$lunora.wire$","bigint","9"]'
    count              -> 0

Storage worked, so the insert that used to throw now succeeded and a balance
check read a confident, wrong zero. Separately, `estimateBytes` still threw on
a bigint and returned `undefined`, which `recordWrite` converts to a hard
BAD_REQUEST — so with a transaction meter attached, i.e. in production, the
write did not merely query wrong, it failed outright.

Encoding decision: STORE THE COMPARABLE SCALAR, do not teach the query side the
tagged form. `encodeDocJson` now projects top-level bigint/bytes fields to a
JSON number / base64 string and parks the wire-tagged originals under a reserved
`__sql__` key; `serializeSqlValue` binds that same projection. Teaching
`serializeSqlValue` the tagged form was the smaller diff but fixes only
equality — a tagged array is still text to SQLite, so ranges, ORDER BY, SUM and
MIN/MAX stay wrong, and wrong silently. The projection makes `json_extract`
return an INTEGER, so all of them are numeric. The parked original is what keeps
the round-trip exact, which `Number(9007199254740993n)` is not; only the SQL
comparison is approximate above 2^53, which is SQLite's own ceiling. Nested
bigint/bytes keep the wire-tagged form — SQL never addresses them.

Backward compatible in both directions: a doc with no bigint/bytes gains no
reserved key and encodes byte-identically (the OCC compare-and-swap depends on
it), a pre-codec plain-JSON row decodes unchanged, and a row written while
bigints were tagged in place still decodes to a bigint.

Also live and fixed:
- `coerceAggregateNumber` refused bigints, so the maintained companion silently
  contributed 0 to a sum over a money column.
- `encodeAggregateKey` threw out of `JSON.stringify` on a bigint `by` field.
- `estimateBytes` threw on bigint (above) and charged an ArrayBuffer as `{}` —
  2 characters for a kilobyte.
- The `cdcSync` admin egress returned DECODED post-images through
  `jsonResponse`, which is plain `Response.json`: a bigint threw a raw TypeError
  and an ArrayBuffer would have reached a consumer's backup as `{}`. It now
  wire-encodes, and `applyCdc` decodes, so export -> replay round-trips.
- `encodeDocJson` re-throws the wire codec's bare TypeError/RangeError as a
  typed BAD_REQUEST instead of an opaque redacted RPC_FAILED.
- The data browser drops the reserved key rather than showing it as a column.

Audited and NOT live: the sql-store `sqliteDecode` bytes branch already exists
and already normalises a pooled view correctly. Byte identity is genuinely false
for array-position `undefined`, NaN, Infinity and Date, but each of those now
round-trips where it previously degraded to null / an ISO string, so the
docblock claim was corrected rather than the behaviour. A `toJSON`-bearing class
instance in a `v.any()` field still cannot be stored — restoring that would mean
changing `shared/wire-codec.ts`, which is a wire-protocol surface; it is now a
typed error naming the value.

Tests: `ctx-db.bigint-bytes.test.ts` now filters, indexes, orders, counts and
aggregates on a bigint column, on a table mirroring `@lunora/payment`'s
`paymentSessions` money columns — the absence of exactly that coverage is what
let this ship. 14 of its 28 cases fail against the previous code; the CDC case
fails with the literal `TypeError: Do not know how to serialize a BigInt`.
Write-throughput benches unchanged (insert 135.3k -> 139.3k hz, patch 56.1k ->
56.0k hz, both inside rme).

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
The `DOC_ORIGINALS_KEY` docblock claimed `__`-prefixed names are reserved by a
codegen column guard. No such guard exists. The only reserved-name enforcement
in the stack is `RESERVED_TABLE_NAMES` (codegen/src/discover-schema.ts:64),
which covers TABLE names colliding with `ctx.db` members, and
`SYSTEM_INDEX_FIELDS` (server/src/schema.ts:861), which is the two-entry list
`["_creationTime", "_id"]` of indexable system fields. There is no
`startsWith("__")` check, no `/^__/`, and no reserved-field list anywhere in
codegen or server.

So a user field named `__sql__` was silently destroyed, on write, two different
ways — the class of defect this whole change exists to remove:

- With a bigint present, `projected[DOC_ORIGINALS_KEY] = originals` overwrote
  the user's value; `{ amount: 10n, __sql__: { keep: "me" } }` read back as
  `{ amount: 10n }`, the field gone for good.
- With no bigint at all — nothing projected, so the key should have been inert —
  `decodeDocJson` still saw `originals !== undefined` and spread the user's own
  object up to the top level, dropping `__sql__`. A document that never went
  near a bigint was corrupted on read.

Both writes succeeded. No throw, no warning.

`encodeDocJson` now rejects the name with a typed BAD_REQUEST. That is the
single chokepoint every write takes (insert, insertMany, patch, replace, soft
delete, CDC append), and it is also what makes `decodeDocJson`'s un-projection
unambiguous: a row carrying a user's `__sql__` can never have been stored. A
field-name guard at schema-definition time was the alternative; it is the larger
change and would have to run on the runtime insert path anyway, which is exactly
where this sits.

The docblock now states what is actually true, with the file:line for both
guards that do exist.

Also pins the migration answer for rows written between ab0afaf and this
branch, which hold a bigint tagged in place rather than projected: they read
correctly but `json_extract` comparisons miss them, and any write through the
store re-projects the whole document. The new test asserts both halves — invisible
to `filter` before a `patch`, found after, with the stored text confirmed to be a
plain number. Rows nobody rewrites need a backfill.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
@netlify

netlify Bot commented Aug 8, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for lunorash ready!

Name Link
🔨 Latest commit 2dfa920
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a7763fa69781000088915eb
😎 Deploy Preview https://deploy-preview-365--lunorash.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Aug 8, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@prisis, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 23 minutes

You've used all free OSS reviews for now. Wait for the free limit to reset to keep reviewing this public repository.

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: d832deb3-49f3-4e4e-b1b3-9e6da939e6fd

📥 Commits

Reviewing files that changed from the base of the PR and between 59b09ae and 2dfa920.

⛔ Files ignored due to path filters (1)
  • packages/shard-engine/__tests__/reprojection-backfill.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (3)
  • packages/shard-engine/src/do-sql.ts
  • packages/shard-engine/src/introspect.ts
  • packages/shard-engine/src/reprojection-backfill.ts

Walkthrough

The PR adds wire encoding for admin RPCs and cursors, SQL-comparable projections for bigint and binary values, document original-value preservation, aggregate validation, and legacy-row reprojection migrations.

Changes

Wire-safe values and SQL projections

Layer / File(s) Summary
SQL-comparable value projection
packages/shard-engine/src/sql-projection.ts, packages/shard-engine/src/serialize-sql.ts, packages/shard-engine/src/index-key-codec.ts, packages/shard-engine/src/estimate-bytes.ts, packages/sql-store/src/value-codec.ts
Bigint values use fixed-width ordered keys. Binary values use Base64 projections. SQL serialization, byte estimation, and validator kind resolution use the shared projection rules.
Document wire-value preservation
packages/shard-engine/src/do-sql.ts, packages/shard-engine/src/introspect.ts
Document encoding stores projected values with originals under __originals__. Decoding and introspection restore valid original values and preserve malformed metadata as ordinary fields.
Query and aggregate boundaries
packages/shard-engine/src/query-args.ts, packages/shard-engine/src/aggregate-tally.ts, packages/shard-engine/src/ctx-db.ts
Cursors use wire encoding. Aggregation accepts safe bigint values, rejects oversized values, decodes aggregate keys, and rejects SQL reduction over projected fields without matching aggregate indexes.
Admin RPC wire envelopes
packages/do/src/shard-do.ts
Admin arguments are decoded before dispatch. Responses use wire-encoded envelopes across migrations, imports, exports, writes, queues, workflows, AI tools, mail, PITR, and introspection operations.
Legacy-row reprojection migrations
packages/shard-engine/src/reprojection-backfill.ts, packages/codegen/src/emit.ts, packages/shard-engine/src/index.ts, packages/do/src/index.ts, packages/cli/src/commands/migrate/handler.ts, packages/payment/docs/index.mdx
The framework detects legacy bigint and byte rows, exposes reserved reprojection migrations, resolves them in generated shards and the CLI, and documents migration commands and verification steps.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant AdminClient
  participant ShardDO
  participant decodeWire
  participant AdminOperation
  AdminClient->>ShardDO: Send encoded admin arguments
  ShardDO->>decodeWire: Decode arguments
  decodeWire-->>ShardDO: Return decoded arguments or BAD_REQUEST
  ShardDO->>AdminOperation: Dispatch operation
  AdminOperation-->>ShardDO: Return operation result
  ShardDO-->>AdminClient: Send encodeWire response envelope
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
Title check ✅ Passed The title clearly identifies the primary change: making shard-engine v.bigint() columns queryable while retaining storage support.
Description check ✅ Passed The description provides a detailed summary, migration impact, reviewer notes, known limitations, linked issues, and comprehensive verification results.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch fix/265-wire-codec-query-parity

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@github-actions

github-actions Bot commented Aug 8, 2026

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@codecov-commenter

codecov-commenter commented Aug 8, 2026 •

Copy link
Copy Markdown

Codecov Report

❌ Patch coverage is 89.61749% with 19 lines in your changes missing coverage. Please review.
⚠️ Please upload report for BASE (alpha@1ca3f16). Learn more about missing BASE report.

Files with missing lines Patch % Lines
packages/do/src/shard-do.ts 76.19% 10 Missing ⚠️
packages/cli/src/commands/migrate/handler.ts 50.00% 3 Missing ⚠️
packages/shard-engine/src/introspect.ts 62.50% 3 Missing ⚠️
packages/shard-engine/src/reprojection-backfill.ts 93.61% 3 Missing ⚠️
Additional details and impacted files
@@           Coverage Diff            @@
##             alpha     #365   +/-   ##
========================================
  Coverage         ?   87.09%           
========================================
  Files            ?     1172           
  Lines            ?    63386           
  Branches         ?    15475           
========================================
  Hits             ?    55204           
  Misses           ?     7663           
  Partials         ?      519           
Files with missing lines Coverage Δ
packages/codegen/src/emit.ts 97.20% <100.00%> (ø)
packages/shard-engine/src/aggregate-tally.ts 100.00% <100.00%> (ø)
packages/shard-engine/src/ctx-db.ts 93.14% <100.00%> (ø)
packages/shard-engine/src/do-sql.ts 96.29% <100.00%> (ø)
packages/shard-engine/src/estimate-bytes.ts 100.00% <100.00%> (ø)
packages/shard-engine/src/index-key-codec.ts 100.00% <ø> (ø)
packages/shard-engine/src/query-args.ts 98.43% <100.00%> (ø)
packages/shard-engine/src/serialize-sql.ts 100.00% <100.00%> (ø)
packages/shard-engine/src/sql-projection.ts 100.00% <100.00%> (ø)
packages/sql-store/src/value-codec.ts 91.89% <100.00%> (ø)
... and 4 more
🚀 New features to boost your workflow:
  • ❄️ Test Analytics: Detect flaky tests, report on failures, and find test suite problems.
  • 📦 JS Bundle Analysis: Save yourself from yourself by tracking and limiting bundle sizes in JS merges.

@codspeed

codspeed Bot commented Aug 8, 2026 •

Copy link
Copy Markdown

Merging this PR will regress 1 benchmark

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
❌ 1 regressed benchmark
✅ 251 untouched benchmarks
⏩ 10 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ findMany — plain table (no scope) 77.8 ms 92.4 ms -15.78%
⚡ indexed: groupBy(projectId) — count (read every counter row) 831.3 µs 421.6 µs +97.18%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing fix/265-wire-codec-query-parity (2dfa920) with alpha (4361cbc)2

Open in CodSpeed

Footnotes

  1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on alpha (1ca3f16) during the generation of this report, so 4361cbc was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

`Number(value)` was the wrong projection, and wrong in the one direction that
is never acceptable: it returned rows that do not match.

`Number` collapses every bigint past 2^53 onto the nearest double, so
9007199254740991n / …992n / …993n all become 9007199254740992. Measured:

    where { externalId: …993n }  -> 2 extra rows (false positives)
    unique index, …789n + …790n  -> second insert throws ConflictError
    .paginate() / orderBy+limit  -> raw TypeError out of the cursor encoder

`origin/alpha` returns [] for all three — uniformly wrong, but in the safe
direction. Trading false negatives for false positives is a regression even
though more queries "work": a `where` is routinely an authorization predicate,
and snowflake ids / epoch-nanos / content hashes are exactly why people reach
for v.bigint(). The unique-index case is worse still — data that persists today
stops persisting, under an error claiming "duplicate" about two different values.

The ceiling comment claiming "SQLite's own numeric stack cannot do better" was
false; SQLite carries exact int64 through json_extract. The lossy step was JS.

## Encoding, decided against the rules in order

1. A query must never return a row that does not match.
2. Exactness beats operation coverage; what cannot be exact fails loudly.
3. Keep SUM/MIN/MAX if possible, but not by accepting a wrong `=`.

A bigint now stores as a fixed-width, sign-aware, order-preserving decimal key:
sign character, then the magnitude zero-padded to 39 digits, nines-complemented
under a lower sign character when negative. Verified against a real SQLite
build: `=` is exact at 2^53 and either side of it, ORDER BY is numeric across a
22-digit range including negatives, and ranges narrow correctly. 39 digits
covers the whole unsigned 128-bit range; past it the write is refused rather
than silently mis-sorted.

Plain decimal text was rejected — exact for `=` but sorts "9" after "10", so
ranges and ORDER BY would return wrong rows instead. Raw JSON integer literals
were rejected too, and are a trap worth recording: SQLite does hold exact int64,
but reading such a row back throws `RangeError: Value is too large to be
represented as a JavaScript number` out of the driver, so the row becomes
unreadable, and the comparison side cannot bind an exact int64 either. Checked
on node:sqlite; NOT checked on workerd's SqlStorage, which no longer matters
since neither candidate survived.

Rule 3 is the cost: padded text is not something SQLite can add up. The
aggregate reader now refuses a v.bigint() field on the scan path rather than
return the 1.5e40 that falls out of coercing it, and the maintained companion
refuses a value its REAL column cannot hold exactly instead of folding a rounded
one into an unauditable total. Declaring an aggregateIndex is the supported way
to sum a bigint column.

## Also fixed

- `.paginate()` / orderBy+limit threw out of `encodeCursor`'s bare
  JSON.stringify — the same defect class this plan is named after, surviving in
  the cursor encoder. Now wire-encoded, and decodeCursor normalises the
  RangeError to the existing typed 400.
- `exportShard` was the un-fixed twin of the cdcSync fix: a bigint column 500'd
  a whole shard export, a bytes column reached a backup as {}. Fixed at the
  boundary instead of per-op — one `adminResponse` encode covers every admin
  handler, one `decodeAdminArgs` covers every ingress (so importShard gets what
  applyCdc had), and the 12-line inline comment is gone. `createShardClient`
  already decodeWires every admin result, and both codecs are the identity on
  pure JSON.
- The `__sql__` docblock claimed a historical invariant it cannot provide. The
  guard is prospective and now says so; decode additionally claims the key only
  when its value is a plain object, so a pre-existing scalar is left alone
  rather than exploded into {"0":"h","1":"e",…}. Renamed to `__originals__`,
  which is what it holds.
- The Studio browser was showing a plausible-but-wrong integer where alpha
  showed an obviously-broken tagged array. It now shows the parked original —
  exact, and already JSON-safe for that read path.
- A bigint past Number.MAX_VALUE projected to Infinity, which is !== undefined,
  so it counted as projected and then got re-tagged by encodeWire. The width
  guard removes the path.

## Structural

The three-branch projection was written twice under a comment asking the copies
to agree. It is now one function (`sql-projection.ts`) that both encodeDocJson
and serializeSqlValue call — the two sides of every comparison are its two call
sites. `??` not `||` at the fallback: 0n and an empty buffer are falsy but real
projections. DOC_ORIGINALS_KEY is declared once and imported. index-key-codec's
docblock said bigints serialize to `String(value)`; corrected, and persisted
read-set keys keep tag "2" (text) exactly as on alpha — the previous commit's
number projection would have moved them to tag "1", this restores that.
serialize-sql no longer overclaims: only bigint and bytes are asserted equal,
and Date/NaN/Infinity are named as still drifting.

Tests: exact `=` above 2^53 across three adjacent values, two distinct large
bigints in a unique index, and a real .paginate() continueCursor round-trip —
7 of the file's 35 cases fail against the previous commit. The gap that let this
through was the same shape as round one: storage was tested above
MAX_SAFE_INTEGER and the query twin never was.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
@prisis
prisis marked this pull request as ready for review August 8, 2026 13:19

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 7

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/do/src/shard-do.ts`:
- Around line 310-314: Update decodeAdminArgs to validate the value returned by
decodeWire before casting or returning it: reject primitives, null, and arrays,
accepting only plain non-array objects. Preserve the existing
LunoraError("BAD_REQUEST", "malformed admin RPC arguments") response for both
decode failures and invalid decoded shapes before admin handlers dispatch.

In `@packages/shard-engine/src/aggregate-tally.ts`:
- Line 191: Update the group-result serialization path around encodeWire and the
groupBy row.key read boundary to decode wire-tagged keys with decodeWire after
JSON.parse. Preserve compatibility by allowing ordinary JSON keys to pass
through unchanged, so declared bigint and byte types are restored in indexed
groupBy results.
- Around line 61-69: Update the aggregate accumulation logic in
aggregate-tally.ts so each updated sum, average numerator, and maintained
minimum/maximum is checked against MAX_SAFE_BIGINT before conversion or
persistence, not just individual bigint inputs. Reuse the existing LunoraError
BAD_REQUEST behavior and preserve exact aggregation by rejecting totals outside
the safe bigint range.

In `@packages/shard-engine/src/ctx-db.ts`:
- Around line 2484-2496: Extract the bigint scan-eligibility validation from the
aggregate() path into a shared helper. Invoke it after each aggregate-index fast
path and before every SQL reducer scan, including the groupBy() scan, so SUM,
AVG, MIN, and MAX on v.bigint() fields throw the same BAD_REQUEST unless a
matching aggregate index handles them.

In `@packages/shard-engine/src/do-sql.ts`:
- Around line 89-94: Update packages/shard-engine/src/do-sql.ts lines 89-94 to
avoid rejecting pre-existing documents solely because they contain
__originals__, while adding schema-level reservation for future documents and
migrating or preflighting affected stored rows. In do-sql.ts lines 142-152,
restore originals only when the versioned internal envelope is present; apply
the same envelope check in introspect.ts lines 903-911 before Studio row
expansion.

In `@packages/shard-engine/src/estimate-bytes.ts`:
- Around line 39-42: Update the byte-handling branch in the estimator around
encodeDocJson’s contract so top-level ArrayBuffer and ArrayBuffer view values
include both the Base64 projection and the preserved wire value under
__originals__. Reuse the same document-encoding representation or sizing logic
for both stored forms, rather than charging only the Base64 payload, while
preserving the existing return behavior.

In `@packages/shard-engine/src/sql-projection.ts`:
- Around line 106-110: Replace the toBase64 usage in sqlComparableProjection()
for ArrayBuffer and ArrayBuffer.isView values with the project’s
byte-order-preserving binary encoder/serializer. Keep the existing handling of
byte offsets and lengths, and ensure indexed byte columns retain lexicographic
byte order.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 7775a822-89ab-4a96-82fb-644d253674a0

📥 Commits

Reviewing files that changed from the base of the PR and between 6c009b9 and 74afd61.

⛔ Files ignored due to path filters (2)
  • packages/do/__tests__/shard-do.admin.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/ctx-db.bigint-bytes.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (10)
  • packages/do/src/shard-do.ts
  • packages/shard-engine/src/aggregate-tally.ts
  • packages/shard-engine/src/ctx-db.ts
  • packages/shard-engine/src/do-sql.ts
  • packages/shard-engine/src/estimate-bytes.ts
  • packages/shard-engine/src/index-key-codec.ts
  • packages/shard-engine/src/introspect.ts
  • packages/shard-engine/src/query-args.ts
  • packages/shard-engine/src/serialize-sql.ts
  • packages/shard-engine/src/sql-projection.ts

Comment thread packages/do/src/shard-do.ts Resolved
Comment on lines +61 to +69
if (typeof value === "bigint") {
if (value > MAX_SAFE_BIGINT || value < -MAX_SAFE_BIGINT) {
throw new LunoraError(
"BAD_REQUEST",
`bigint ${value.toString()} exceeds Number.MAX_SAFE_INTEGER and cannot be aggregated exactly — aggregate a narrower column, or read the rows and reduce them in the handler`,
);
}

return Number(value);

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | ⚡ Quick win

Validate the accumulated aggregate value.

This check validates each bigint input but not the running total. 9007199254740991n and 2n both pass, but their sum cannot be represented exactly as a JavaScript number. The companion REAL value can silently round.

Validate each updated sum, average numerator, and maintained extreme before persisting it.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/aggregate-tally.ts` around lines 61 - 69, Update
the aggregate accumulation logic in aggregate-tally.ts so each updated sum,
average numerator, and maintained minimum/maximum is checked against
MAX_SAFE_BIGINT before conversion or persistence, not just individual bigint
inputs. Reuse the existing LunoraError BAD_REQUEST behavior and preserve exact
aggregation by rejecting totals outside the safe bigint range.

Comment thread packages/shard-engine/src/ctx-db.ts Resolved
Comment on lines +89 to +94
if (Object.hasOwn(document, DOC_ORIGINALS_KEY)) {
throw new LunoraError(
"BAD_REQUEST",
`"${DOC_ORIGINALS_KEY}" is reserved by the row store's document encoding and cannot be used as a field name — rename the field`,
);
}

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🗄️ Data Integrity & Integration | 🟠 Major | 🏗️ Heavy lift

Use an unambiguous metadata format for projected originals.

__originals__ was valid user data before this change. The new format cannot distinguish a legacy user object from internal metadata.

  • packages/shard-engine/src/do-sql.ts#L89-L94: do not make pre-existing documents unwritable solely because they use this field name.
  • packages/shard-engine/src/do-sql.ts#L142-L152: restore originals only when a versioned internal envelope is present.
  • packages/shard-engine/src/introspect.ts#L903-L911: apply the same envelope check before expanding rows for Studio.

Add schema-level reservation for future documents and migrate or preflight affected stored rows.

📍 Affects 2 files
  • packages/shard-engine/src/do-sql.ts#L89-L94 (this comment)
  • packages/shard-engine/src/do-sql.ts#L142-L152
  • packages/shard-engine/src/introspect.ts#L903-L911
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/do-sql.ts` around lines 89 - 94, Update
packages/shard-engine/src/do-sql.ts lines 89-94 to avoid rejecting pre-existing
documents solely because they contain __originals__, while adding schema-level
reservation for future documents and migrating or preflighting affected stored
rows. In do-sql.ts lines 142-152, restore originals only when the versioned
internal envelope is present; apply the same envelope check in introspect.ts
lines 903-911 before Studio row expansion.

Comment on lines +106 to +110
if (value instanceof ArrayBuffer) {
return toBase64(new Uint8Array(value));
}

return ArrayBuffer.isView(value) ? toBase64(new Uint8Array(value.buffer, value.byteOffset, value.byteLength)) : undefined;

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

🧩 Analysis chain

🏁 Script executed:

#!/bin/bash
set -euo pipefail

fd -a '^base64\.(ts|tsx|js|jsx)$' shared | while IFS= read -r file; do
  echo "== $file =="
  sed -n '1,240p' "$file"
done

rg -n -C 3 'sqlComparableProjection|toBase64|v\.bytes|orderBy|withIndex' packages/shard-engine shared

Repository: anolilab/lunora

Length of output: 50371


🏁 Script executed:

#!/bin/bash
set -euo pipefail

echo "== sql-projection outline =="
ast-grep outline packages/shard-engine/src/sql-projection.ts || true

echo "== sql-projection relevant lines =="
sed -n '1,180p' packages/shard-engine/src/sql-projection.ts

echo "== query-args toBase64 =="
sed -n '1,180p' packages/shard-engine/src/query-args.ts

echo "== bytes index usage =="
python3 - <<'PY'
from pathlib import Path
matches = []
for p in Path("packages/shard-engine/src").rglob("*.ts"):
    text = p.read_text()
    if "sqlComparableProjection" in text or "toBase64" in text:
        matches.append(p)
for m in matches:
    print(m)
PY

echo "== small behavior probe for btoa/std base64 order for relevant byte patterns =="
node - <<'JS'
function toBase64(bytes) {
  let binary = "";
  const chunk = 0x8000;
  for (let index = 0; index < bytes.length; index += chunk) binary += String.fromCharCode(...bytes.subarray(index, index + chunk));
  return btoa(binary);
}
const cases = [
  [0xd0],
  [0x00],
  [0x0d, 0x00],
  [0xd0, 0x00],
  [0x0d, 0x01],
  [0xd0, 0x01],
];
for (const bytes of cases) console.log(JSON.stringify(Object.values(bytes).map(b => b.toString(16).padStart(2,'0').toUpperCase())), "=>", toBase64(new Uint8Array(bytes)));
console.log("0xd0 < 0x00 byte order:", toBase64(new Uint8Array([0xd0])) < toBase64(new Uint8Array([0x00])));
console.log("0xd0 encodes to same prefix plus padded suffix as 0x0d when length 1:", toBase64(new Uint8Array([0xd0])).startsWith("Dc="), toBase64(new Uint8Array([0x0d])).slice(0,3));
JS

Repository: anolilab/lunora

Length of output: 13618


Do not use Base64 for byte order preservation.

sqlComparableProjection() stores v.bytes() values with shared/base64.toBase64(), whose standard Base64 alphabet does not preserve byte order; for example 0xd0 encodes to 0A==, which sorts before 0x00 encoded as AA==. Use a byte-preserving binary encoder/serializer instead of Base64 for indexed byte columns.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/sql-projection.ts` around lines 106 - 110, Replace
the toBase64 usage in sqlComparableProjection() for ArrayBuffer and
ArrayBuffer.isView values with the project’s byte-order-preserving binary
encoder/serializer. Keep the existing handling of byte offsets and lengths, and
ensure indexed byte columns retain lexicographic byte order.

Rows written between ab0afaf and the SQL-comparable projection stored
v.bigint()/v.bytes() as a wire-tagged array left in place at `$.field`. They
read back correctly, but the stored text is an array, so `json_extract` never
matches: filter/withIndex miss them, ORDER BY sorts them as text, SUM counts
them as 0.

The projection fix heals a row on any patch/replace and migrates nothing on its
own, so what was left is the rows nobody rewrites — for `paymentSessions`,
precisely the settled sessions, which are the ones most likely to be queried by
amount and least likely to be written again.

No runner here, deliberately: `runDataMigration` already provides keyset
batching through the normal writer, resumability, idempotence, dryRun, and the
property this actually needs — cursor stability, since `replace` preserves
`_id`/`_creationTime` so a rewritten row never moves relative to the cursor.
This adds only what is specific to the defect:

- Affected-table derivation from the schema. A table with no bigint/bytes column
  cannot hold an affected row and is never scanned; `.global()` tables are
  excluded outright, their rows live in the sql-store backend.
- A detector over the STORED TEXT, because a decoded document cannot say which
  format it came from — that is the whole defect. The signal is the wire
  sentinel at element 0 of a top-level projected field, so `json_extract`
  returns NULL for a current row and the sentinel for a legacy one. Narrow on
  purpose: a predicate matching everything would rewrite every shard on every
  run and would look like it was working. Paths are bound, not interpolated.
- An IDENTITY transform. Returning the document unchanged makes the runner
  rewrite it through `replace`, and `encodeDocJson` re-projects on the way down.
  Nothing reconstructs a value, so there is no third implementation of the
  projection to drift — which is what produced the original defect. A raw SQL
  UPDATE over `__doc__` would be faster and would bypass triggers, CDC and
  subscriber notification.
- Rows already in the current projection return `undefined`, so the runner
  counts them processed and leaves them alone. Rewriting them is wasted work and
  pokes every subscriber on the row.

The sentinel is derived from `encodeWire(0n)[0]` rather than copied: the codec
does not export it and must not be edited, and a hand-copied literal would be a
second definition free to drift.

Invocation needs no CLI change — `lunora migrate up/status <id>` already takes an
id, fans out per shard, and has --dry-run/--batch-size/--steps. Codegen resolves
the reserved `__lunora_reproject__<table>` id when the app's own registry misses,
built per-invocation because the detector needs the shard's `sql` handle.
`ensureMigrated()` moved above the lookup so the tables exist first.

Tests cover the plan's list plus the contract: a seeded legacy row becomes
queryable and stores the current projection, a current row is not rewritten, a
mixed table converges, a resumed run does not double-count, dryRun changes
nothing, v.bytes() alongside v.bigint(), a nested tagged value is NOT matched,
and a decoded document with Date/Map/Set/NaN/Infinity/array-undefined/nested
bigint is byte-identical across the rewrite.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 2

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/payment/docs/index.mdx`:
- Around line 388-392: Update the migration instructions near the
`__lunora_reproject__<table>` identifier to limit repetition to affected
non-global tables. Explicitly exclude tables declared with `.global()`, while
retaining the existing bigint/bytes column criteria.

In `@packages/shard-engine/src/reprojection-backfill.ts`:
- Around line 91-95: Update legacyRowPredicate so each field name is escaped
according to SQLite JSON-path rules before constructing the bound path
parameter. Preserve the existing $.<field>[0] shape and matching behavior while
ensuring names containing dots or other path-significant characters are treated
as a single top-level key.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 3b2efc29-2312-4852-a5ad-ff829161aeb0

📥 Commits

Reviewing files that changed from the base of the PR and between 74afd61 and bb398f4.

⛔ Files ignored due to path filters (18)
  • api-snapshots/do.api.md is excluded by none and included by none
  • api-snapshots/lunora.api.md is excluded by none and included by none
  • api-snapshots/shard-engine.api.md is excluded by none and included by none
  • examples/auth-playground/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/blog/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/chess/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/expo/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/feedback-board/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/kanban-board/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/notify-demo/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/offline-rejections/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/payment-demo/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/realtime-cursors/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/tanstack-start/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/team-chat/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/todo-app/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • packages/codegen/__tests__/fixtures/simple/expected/_generated/shard.ts is excluded by !**/_generated/**, !**/__tests__/** and included by packages/**
  • packages/shard-engine/__tests__/reprojection-backfill.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (5)
  • packages/codegen/src/emit.ts
  • packages/do/src/index.ts
  • packages/payment/docs/index.mdx
  • packages/shard-engine/src/index.ts
  • packages/shard-engine/src/reprojection-backfill.ts

Comment thread packages/payment/docs/index.mdx Resolved
prisis added a commit that referenced this pull request Aug 8, 2026
Completed plans do not stay in this directory - the record is the README
row and git history. 311 shipped inside #365, so its file would have
been born and deleted in a day. Its one live thread, wiring
countLegacyRows into doctor, moves into the row so it does not vanish
with the file.

312 stays: its design decisions need a measurement first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
One root cause under four HIGH findings: `encodeDocJson` projects on the value's
runtime type (`typeof value === "bigint"`), while every guard and the whole
backfill dispatched on the declared validator `kind`. `v.optional(v.bigint())`
has `kind === "optional"` with the real validator on `_meta.inner`, so those two
disagreed, and each disagreement was a silent wrong answer:

- `aggregate` sum over an optional bigint reduced the padded keys — 100n + 250n
  read as 2e+39, no throw.
- `groupBy`'s scan path had no guard at all, so a PLAIN bigint did the same, and
  its group keys came back as 40 characters of zero padding.
- `reprojectableFields` returned [] for an optional bigint, so `countLegacyRows`
  read 0 and `buildReprojectionMigration` returned undefined. The docs tell an
  operator that a second --dry-run reading 0 means nothing is left, so an
  affected app was told it was clean — a false all-clear on a data-migration
  completeness check.

The repo already owned the correct predicate (`effectiveColumnKind` in
sql-store, added for this same plan). It now lives once in
`shared/effective-kind` — neither store depends on the other, and a fourth copy
is what caused this. sql-store's export becomes a thin alias.

`isProjectedKind` / `mayHoldProjectedValue` build on it: the guards use the
narrow one, the backfill the wide one, which also counts `v.any()`/`v.union()`/
`v.from()` since an untyped field can hold a bigint the projection then projects.
Widening the backfill is only safe because the row predicate now tests the tagged
value's TYPE slot rather than merely the sentinel — a `Date` under a `v.any()`
field is legitimately tagged today, and matching it would rewrite the row, re-tag
the Date, and leave it "legacy" forever.

Also fixed:

- The backfill was unreachable. `resolveMigrationTable` AST-scans the user's
  `lunora/` for `defineMigration({ id })`, so a reserved framework id exited 1
  with no network call. The CLI now derives the table from the id. The prefix
  moved to `shared/reprojection-id` because the CLI depends on neither the engine
  nor `@lunora/do`, and spelling it twice would surface as MIGRATION_NOT_FOUND
  with nothing pointing at the cause.
- Studio grid regression. `adminResponse` encodes every admin result and
  `expandDocumentRows` spread ALREADY-tagged originals, so the codec escaped them
  and the grid rendered the tag instead of `4200n`. It decodes first now, which
  fixes the POST and WS paths together. Two comments asserted the opposite of
  what the code did — one still described the world before `adminResponse`
  existed; both rewritten.
- One oversize legacy row wedged the backfill forever: the legacy format stored a
  bigint of any width (a uint256 balance is 78 digits, and this repo ships
  `@lunora/x402`), the transform threw, and the run persisted `failed` with the
  cursor pinned so every resume died identically. It skips and reports now.
- Indexed `groupBy` returned the tagged array as the group key (`encodeAggregateKey`
  writes `encodeWire`, the reader was a bare `JSON.parse`) — on alpha this threw,
  so it was a loud failure turned into a wrong answer.
- `estimateBytes` undercounted by ~6x for bigint and ~2x for bytes (measured),
  because a projected column is stored TWICE. Both copies are charged now, and
  the docblock states the ~4x/~2x on-disk cost the branch never mentioned.
- A field named `__proto__` lost its value to `Object.prototype`'s setter; the
  originals map is null-prototype.
- Comments corrected where they overclaimed: base64 is NOT order-preserving, so
  `ORDER BY` on a `v.bytes()` column is ordered by base64 text (equality and the
  backfill are unaffected); the aggregate guard bounds each contribution, not the
  running total, which rounds in the REAL column exactly as a `v.number()` sum
  does; binding a JSON path defeats SQL injection but not SQLite's path grammar,
  so field names are quoted; 39 digits clears uint128's ~3.4e38, not 1.7e38.

From CodeRabbit, additionally: admin args are re-checked for object-ness after
decode (a tagged `undefined` or a root array changes the SHAPE, and handlers
index before validating), and the payment docs no longer send operators at
`.global()` tables, which correctly answer MIGRATION_NOT_FOUND.

Tests: an ordering property test over 211 values (~44.5k pairs) asserting
`sign(key(a) cmp key(b)) === sign(a - b)`, checked in JS and again under a real
SQLite ORDER BY; the 39-digit refusal; `v.optional(v.bigint())` end to end. Fixed
three broken ones — a "no-op on a second full run" that never ran twice, a
"groups by a bigint column" that called `count`, and four assertions comparing
the projection against itself via `serializeSqlValue`, which would have passed
had the projection reverted to `String(value)`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/shard-engine/src/ctx-db.ts`:
- Around line 962-965: Update assertReducibleBySql to use
mayHoldProjectedValue(validator) instead of only checking exact projected kinds,
rejecting broad validators such as unions, any, or from-based validators that
may contain runtime bigint or bytes values while preserving the existing
SQL-reducibility validation behavior.
🪄 Autofix

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 78caab4f-bcb9-4bcc-8317-a5ef3d999501

📥 Commits

Reviewing files that changed from the base of the PR and between bb398f4 and 59b09ae.

⛔ Files ignored due to path filters (5)
  • packages/shard-engine/__tests__/ctx-db.bigint-bytes.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/reprojection-backfill.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/sql-projection.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • shared/effective-kind.ts is excluded by none and included by none
  • shared/reprojection-id.ts is excluded by none and included by none
📒 Files selected for processing (13)
  • packages/cli/src/commands/migrate/handler.ts
  • packages/cli/tsconfig.json
  • packages/do/src/shard-do.ts
  • packages/payment/docs/index.mdx
  • packages/shard-engine/src/aggregate-tally.ts
  • packages/shard-engine/src/ctx-db.ts
  • packages/shard-engine/src/do-sql.ts
  • packages/shard-engine/src/estimate-bytes.ts
  • packages/shard-engine/src/index-key-codec.ts
  • packages/shard-engine/src/introspect.ts
  • packages/shard-engine/src/reprojection-backfill.ts
  • packages/shard-engine/src/sql-projection.ts
  • packages/sql-store/src/value-codec.ts
🚧 Files skipped from review as they are similar to previous changes (7)
  • packages/shard-engine/src/index-key-codec.ts
  • packages/shard-engine/src/aggregate-tally.ts
  • packages/payment/docs/index.mdx
  • packages/shard-engine/src/do-sql.ts
  • packages/shard-engine/src/introspect.ts
  • packages/shard-engine/src/estimate-bytes.ts
  • packages/do/src/shard-do.ts

Comment on lines +962 to +965
const assertReducibleBySql = (definition: TableDefinitionLike, field: string, label: string): void => {
const validator = definition.shape[field];

if (validator && isProjectedKind(validator)) {

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

🎯 Functional Correctness | 🟠 Major | ⚡ Quick win

Reject scan aggregates for validators that can hold projected values.

Lines 963-965 only reject exact bigint and bytes validators. A v.union(), v.any(), or v.from() field can contain a runtime bigint or byte value. That value is stored as a projection, but the aggregate and groupBy() scan paths then reduce or group the projection key.

Use mayHoldProjectedValue() here, or add an equivalent safe scan check. This prevents SUM, MIN, MAX, and GROUP BY from returning projection keys for broad validators.

Proposed fix
-import { isProjectedKind } from "./sql-projection";
+import { mayHoldProjectedValue } from "./sql-projection";
...
-    if (validator && isProjectedKind(validator)) {
+    if (validator && mayHoldProjectedValue(validator)) {
📝 Committable suggestion

‼️ IMPORTANT
Carefully review the code before committing. Ensure that it accurately replaces the highlighted code, contains no missing lines, and has no issues with indentation. Thoroughly test & benchmark the code to ensure it meets the requirements.

Suggested change
const assertReducibleBySql = (definition: TableDefinitionLike, field: string, label: string): void => {
const validator = definition.shape[field];
if (validator && isProjectedKind(validator)) {
const assertReducibleBySql = (definition: TableDefinitionLike, field: string, label: string): void => {
const validator = definition.shape[field];
if (validator && mayHoldProjectedValue(validator)) {
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/ctx-db.ts` around lines 962 - 965, Update
assertReducibleBySql to use mayHoldProjectedValue(validator) instead of only
checking exact projected kinds, rejecting broad validators such as unions, any,
or from-based validators that may contain runtime bigint or bytes values while
preserving the existing SQL-reducibility validation behavior.

Two differences from `shared/json-path-segment.ts` on the branch that fixes this
repo-wide, both now removed:

- `\` was not escaped, so a backslash-bearing field yielded `$."back\slash"`,
  which SQLite 3.53 resolves to NULL rather than erroring — the same silent miss
  the quoting exists to prevent. It is escaped BEFORE `"`, or the backslash the
  quote escape inserts gets doubled in turn.
- It quoted unconditionally; it now emits the bare form for a field matching the
  identifier regex `parse-validator.ts` already enforces on every schema field.

Neither was reachable here — the backfill's field names come from the schema
shape and codegen constrains those to identifiers — so this is drift prevention,
not a bug fix. Two copies of an escaping rule diverging in opposite directions is
the defect the shared helper exists to fix, and one copy being unreachable today
is how that survives to become reachable later.

Kept local rather than imported: the shared file exists only on the other branch,
and importing it would turn a soft merge-order note into a hard dependency
between two otherwise independent PRs. Whichever lands second deletes this copy,
and because the two now agree exactly, that deletion is a no-op refactor.

`jsonPathSegment` is exported from the module for its test only, deliberately not
re-exported from the barrel, so the public API surface is unchanged.

Tests cover both differences plus a SQLite round-trip proving the emitted path
actually resolves — all three fail against the previous implementation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP
prisis added a commit that referenced this pull request Aug 8, 2026
* docs(plans): file the two follow-ups from the bigint fix

311 is the data the defect left behind: rows written between ab0afaf
and #365 hold the bigint tagged in place, so they read correctly but no
json_extract comparison matches them. #365 heals a row on any write and
migrates nothing, which leaves exactly the rows nobody rewrites - for
paymentSessions, the settled sessions. runDataMigration already gives
resumability, idempotence and cursor stability, so the work is a
transform and a detector, not a runner.

312 is the price #365 paid. A soft-delete table can never reach the
aggregate companion, and declaring aggregateIndex on a bigint column
caps that column's writes at 2^53. Both throw rather than return a
wrong answer, and closing either by rounding is a STOP condition.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP

* docs(plans): record 311 as implemented on the 365 branch

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP

* docs(plans): sync the index with 311 shipping inside 365

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP

* docs(plans): retire 311, keep 312

Completed plans do not stay in this directory - the record is the README
row and git history. 311 shipped inside #365, so its file would have
been born and deleted in a day. Its one live thread, wiring
countLegacyRows into doctor, moves into the row so it does not vanish
with the file.

312 stays: its design decisions need a measurement first.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_011sk9BLaUZDkPVAZ1sKhuDP

---------

Co-authored-by: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — 2dfa9202 Deployed Aug 8, 2026 by prisis via Benchmarks #1561
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants