Skip to content

feat(shard-engine): range-precise invalidation, tx limits, batched global reads - #239

Merged
prisis merged 12 commits into
alphafrom
feat/range-invalidation
Jul 31, 2026
Merged

prisis merged 12 commits into
alphafrom
feat/range-invalidation

Conversation

@prisis

@prisis prisis commented Jul 30, 2026 •

Copy link
Copy Markdown
Member

1. Range-precise query invalidation

ctx.db.query(table) stamped its dependency eagerly, before the chain revealed whether .withIndex(...) was coming, so it had to assume the whole table. The codebase already named the fix in a comment:

"we stamp the safe upper bound (*scan). Future refinement would push the hook into buildReader's terminal runFetch so an indexed read can record per-id deps."

The dependency now comes from the reader's terminal, where the staged plan is final, and an indexed read records the contiguous slice it touched. A write to channelId: "B" no longer wakes every subscriber reading channelId: "A".

  • index-key-codec.ts — order-preserving encoding matching SQLite's NULL < INTEGER/REAL < TEXT. Text encodes as UTF-8 bytes because JS compares UTF-16 code units and SQLite's BINARY compares UTF-8 — they disagree on astral characters, and there's a test pinning it.
  • read-write-set.ts — half-open [lo, hi) slices. Half-open is what makes exclusive bounds correct on compound indexes: (a=1, b=5) sorts above the bare bound enc(1)!enc(5), so gt("b", 5) must lift lo past every row with b === 5.
  • read-footprint.ts / subscription-range-gate.ts — the refresh gate, extracted from ShardDO because it is pure and its rules are the subtlest part of this.

Narrowing is an optimization, never a correctness input. An unencodable value, an unmodelled comparator, an unknown write position, an index the write produced no key for, or a table read in more than one way all fall back to whole-table invalidation. Missing an invalidation serves a live subscriber stale data forever; a surplus one costs a re-run.

2. Per-transaction resource limits

A Durable Object is single-threaded with a hard memory ceiling, so one unbounded mutation took the whole shard down and the only diagnostic was an isolate that vanished. Reads and writes are now charged against ceilings (32 MiB written / 50k rows written / 100k rows read) and the transaction stops with a new TRANSACTION_LIMIT_EXCEEDED naming the ceiling it hit.

⚠️ This is a new hard failure mode. Workloads that previously just ran slowly can now fail. The ceilings are overridable via the transactionLimits() seam on the generated shard class.

3. Coalesced point reads on the global path

Promise.all(ids.map(ctx.db.get)) paid one network round-trip per id against .global(). Same-tick reads now collapse into a single IN (...). Landed in @lunora/sql-store, so it covers D1 and Hyperdrive at once.

4. Vnode placement ring (dark)

Routing groundwork for progressive sharding: fixed vnode ring, FNV-1a + jump consistent hash, directory. Bootstraps at shardCount: 0, where placement is byte-identical to today's single-DO topology. Rebalance, the WAL/watermark protocol, and autoscale are deliberately absent — each needs its own design, and a half-built version risks misrouting live rows. Nothing consumes it, so it stays off the package barrel.

Review notes

Both /thermos passes were run and every finding is fixed. Three were serious and worth calling out, since all three were bugs this branch introduced:

  • The refresh gate was inert and unsafe. It read ranges from a per-request field on the DO that only runCachedQuery set — and runCachedQuery has no production callers, and the field was never reset. Any subscription would have inherited an unrelated request's slices. Subscriptions now report their own read footprint; the shared field is gone.
  • A patch that moved a row between slices never woke the slice it left. The delta carried only the post-image. Write positions are now computed once on the write path across both images and carried on the delta — which also removed a second, divergent derivation.
  • Subscription re-runs spent the writing mutation's resource budget, and a trip surfaced as a swallowed catch that silently dropped a live update. Each re-run now gets its own budget.

Also fixed: an insertManyUnsafe path that wrote the whole batch before metering; a name collision with @lunora/platform's existing ShardDirectory/resolveShard; and a dep-encoding scheme where a document id shaped like a range could have forged one (ranges now travel as objects on their own channel, so nothing parses user data).

Not addressed: the quality pass recommends splitting this into four PRs. Kept as one branch with four logically separated commits; happy to split if you'd rather review them independently.

Verification

  • 2,560 tests pass — shard-engine 830, do 501, codegen 939, sql-store 87, errors 203 (58 new).
  • tsc --noEmit clean, ESLint clean, Prettier clean, package.json sort clean, API snapshots regenerated and matching.
  • Generated artifacts regenerated: the codegen golden fixture and all 8 examples' _generated/shard.ts.
  • Backward compatible: recordChangedTable and invalidate gained optional parameters, so already-generated code and third-party ShardDO subclasses keep compiling with the old conservative behaviour.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Added configurable transaction limits with actionable errors when thresholds are exceeded.
    • Added more consistent shard placement and indexed range support.
  • Performance
    • Batched same-tick point reads to reduce database queries.
  • Bug Fixes
    • Improved subscription refresh accuracy by detecting relevant indexed changes.
    • Enhanced cache invalidation when records move between indexed ranges.
    • Improved handling of indexed reads and updates across affected records.

@netlify

netlify Bot commented Jul 30, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for lunorash ready!

Name Link
🔨 Latest commit 7e84ede
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a6c62b6c63aa10008c0be39
😎 Deploy Preview https://deploy-preview-239--lunorash.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Jul 30, 2026 •

Copy link
Copy Markdown

Review Change Stack

Walkthrough

The PR adds indexed read tracking, range-aware cache and subscription invalidation, transaction resource limits, vnode-based shard placement utilities, and microtask-batched SQL point reads.

Changes

Range-aware runtime

Layer / File(s) Summary
Index range and read-footprint contracts
packages/shard-engine/src/read-write-set.ts, packages/shard-engine/src/index-key-codec.ts, packages/shard-engine/src/read-footprint.ts, packages/shard-engine/src/schema-types.ts, packages/shard-engine/src/index.ts
Adds order-preserving index encoding, half-open range algebra, read footprints, mutation index metadata, and public exports.
Transaction budget enforcement
packages/shard-engine/src/transaction-headroom.ts, packages/errors/src/catalog.ts, packages/shard-engine/src/estimate-bytes.ts, packages/shard-engine/src/ctx-db.ts, packages/do/src/shard-do.ts
Tracks read rows, written rows, and written bytes. Raises TRANSACTION_LIMIT_EXCEEDED when configured ceilings are exceeded.
Indexed database and cache invalidation
packages/shard-engine/src/ctx-db.ts, packages/shard-engine/src/reactive-cache.ts, packages/shard-engine/src/ctx-db-companions.ts
Reports indexed read ranges, meters database operations, propagates before-and-after index keys, and invalidates range-dependent cache entries.
Subscription refresh gating and generated wiring
packages/do/src/shard-do.ts, packages/codegen/src/emit.ts, packages/do/src/index.ts, packages/shard-engine/src/subscription-range-gate.ts
Carries read ranges and changed index keys through generated contexts and coalesced refreshes. Skips subscriptions whose recorded ranges do not overlap writes.

Shard placement utilities

Layer / File(s) Summary
VNode hashing and placement
packages/shard-engine/src/shard-ring.ts
Adds vnode directory bootstrapping, hashing, system-table detection, document placement, vnode enumeration, and tier-zero detection.

SQL point-read batching

Layer / File(s) Summary
Coalesced SQL point reads
packages/sql-store/src/point-read-batcher.ts, packages/sql-store/src/ctx-db.ts
Coalesces same-tick point reads, shares duplicate waiters, chunks large requests, preserves missing rows, and routes get through the batcher.

Estimated code review effort: 4 (Complex) | ~60 minutes

Sequence Diagram(s)

sequenceDiagram
  participant ShardWrite
  participant CtxDb
  participant ReactiveCache
  participant SubscriptionRefresh
  ShardWrite->>CtxDb: write row and derive index keys
  CtxDb->>ReactiveCache: invalidate overlapping index ranges
  CtxDb->>SubscriptionRefresh: publish changed tables and index keys
  SubscriptionRefresh->>SubscriptionRefresh: compare writes with recorded read ranges
  SubscriptionRefresh-->>ShardWrite: refresh affected subscriptions
Loading

Possibly related PRs

  • anolilab/lunora#37: The PR adds related subscription, write-refresh, and generated shard infrastructure.
  • anolilab/lunora#70: Both PRs modify subscription refresh handling in packages/do/src/shard-do.ts.
  • anolilab/lunora#125: Both PRs modify subscription synchronization and local mirror update handling.
🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly summarizes the primary changes: range-precise invalidation, transaction limits, and batched global reads.
Description check ✅ Passed The description provides a detailed summary, review notes, verification results, compatibility details, and known limitations.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch feat/range-invalidation

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@github-actions

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 8

🧹 Nitpick comments (11)
packages/shard-engine/src/shard-ring.ts (2)

78-88: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Hoist TextEncoder out of the hot path.

A new TextEncoder is allocated on every fnv1a64 call. Since this function underlies every placement resolution and will be on the routing hot path once wired in, reuse a single module-level encoder instance instead.

⚡ Proposed fix
+const textEncoder = new TextEncoder();
+
 const fnv1a64 = (input: string): bigint => {
     let hash = FNV_OFFSET_BASIS;
-    const bytes = new TextEncoder().encode(input);
+    const bytes = textEncoder.encode(input);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/shard-ring.ts` around lines 78 - 88, Move the
TextEncoder allocation out of fnv1a64 and create one module-level encoder
instance for reuse. Update fnv1a64 to encode input through that shared instance
while preserving the existing hashing behavior.

129-150: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Duplicate positive-integer validation.

bootstrapDirectory (Lines 130-132) and vnodeForId (Lines 145-147) repeat the same Number.isInteger(...) && > 0 check with near-identical error messages. As per path instructions, "Follow DRY principles". Extracting a shared assertPositiveInteger(value, name) helper would remove the duplication and guarantee consistent error text.

♻️ Proposed fix
+const assertPositiveInteger = (value: number, name: string): void => {
+    if (!Number.isInteger(value) || value <= 0) {
+        throw new RangeError(`${name} must be a positive integer, received ${String(value)}`);
+    }
+};
+
 const bootstrapDirectory = (vnodeCount: number = DEFAULT_VNODE_COUNT, shardCount = 0): VnodeDirectory => {
-    if (!Number.isInteger(vnodeCount) || vnodeCount <= 0) {
-        throw new RangeError(`vnodeCount must be a positive integer, received ${String(vnodeCount)}`);
-    }
+    assertPositiveInteger(vnodeCount, "vnodeCount");

     if (!Number.isInteger(shardCount) || shardCount < 0) {
         throw new RangeError(`shardCount must be a non-negative integer, received ${String(shardCount)}`);
     }
     ...
 };

 const vnodeForId = (id: string, vnodeCount: number): number => {
-    if (!Number.isInteger(vnodeCount) || vnodeCount <= 0) {
-        throw new RangeError(`vnodeCount must be a positive integer, received ${String(vnodeCount)}`);
-    }
+    assertPositiveInteger(vnodeCount, "vnodeCount");

     return jumpConsistentHash(id, vnodeCount);
 };
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/shard-ring.ts` around lines 129 - 150, Extract the
duplicated positive-integer validation from bootstrapDirectory and vnodeForId
into a shared assertPositiveInteger(value, name) helper. Have the helper throw
the consistent RangeError message and invoke it for vnodeCount in both
functions, preserving the existing validation behavior.

Source: Path instructions

packages/shard-engine/src/index-key-codec.ts (1)

93-94: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Hoist the TextEncoder to module scope.

encodeString runs once per indexed text field per written row and per range bound; a shared encoder avoids a fresh allocation on that hot path.

♻️ Proposed refactor
+const TEXT_ENCODER = new TextEncoder();
+
 /** UTF-8 bytes of `value` as lowercase hex — byte order == SQLite `BINARY` order. */
 const encodeString = (value: string): string => {
-    const bytes = new TextEncoder().encode(value);
+    const bytes = TEXT_ENCODER.encode(value);
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/index-key-codec.ts` around lines 93 - 94, Hoist a
single TextEncoder instance to module scope and update encodeString to reuse it
instead of constructing a new encoder on each call. Preserve the existing
string-to-byte encoding behavior.
packages/shard-engine/src/read-write-set.ts (1)

295-313: 🚀 Performance & Scalability | 🔵 Trivial | 💤 Low value

Consider indexing keys by index name once instead of filter per range.

keysTouchRanges runs per subscription per flush, and a coalesced batch can carry many keys; the inner keys.filter allocates an array for every range. A single Map<string, IndexKeyEntry[]> built once removes the quadratic allocation.

♻️ Proposed refactor
-    return ranges.some((range) => {
-        const matching = keys.filter((entry) => entry.index === range.index);
-
-        if (matching.length === 0) {
-            return true;
-        }
-
-        return matching.some((entry) => rangeContains(range, entry));
-    });
+    const byIndex = new Map<string, IndexKeyEntry[]>();
+
+    for (const entry of keys) {
+        const bucket = byIndex.get(entry.index);
+
+        if (bucket) {
+            bucket.push(entry);
+        } else {
+            byIndex.set(entry.index, [entry]);
+        }
+    }
+
+    return ranges.some((range) => {
+        const matching = byIndex.get(range.index);
+
+        // A range over an index we computed no key for is unprovable → touched.
+        return matching === undefined || matching.some((entry) => rangeContains(range, entry));
+    });
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/read-write-set.ts` around lines 295 - 313, Refactor
keysTouchRanges to build a single Map keyed by index name from keys before
iterating ranges, then retrieve each range’s matching entries from that Map
instead of calling keys.filter per range. Preserve the existing behavior for
missing indexes and rangeContains checks.
packages/codegen/src/emit.ts (2)

4286-4317: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

De-duplicate the two databaseOptions templates.

The vector and non-vector variants now differ only by the onWrite, line, yet every option added in this PR had to be edited twice — exactly the drift risk DRY guards against. Build one template with a conditional field.

♻️ Sketch
-    const databaseOptions = hasVectors
-        ? `{${authField}
-                broadcast: (delta) => {
-...
-            }`
-        : `{${authField}
-...
-            }`;
+    const onWriteField = hasVectors ? "\n                onWrite," : "";
+    const databaseOptions = `{${authField}
+                broadcast: (delta) => {
+                    this.recordChangedTable(delta.table, delta.indexKeys);
+                },
+                cdc: config.cdc ?? false,
+                enforceRls: true,
+                headroom: this.transactionHeadroom(),
+                onIndexUse: this.getCtxDbIndexUseHook(),
+                onRead: options.onRead ?? this.getCtxDbReadHook(),
+                onReadRange: options.onReadRange,${onWriteField}
+                scheduler,
+                schema: schema as unknown as SchemaLike,
+                sql: this.sql as SqlExec,
+                storage,${globalDatabaseField}
+            }`;

Note the emitted key order must stay sorted for the package-sort/API-snapshot checks.

As per path instructions, "Follow DRY principles".

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/codegen/src/emit.ts` around lines 4286 - 4317, Deduplicate the two
databaseOptions template branches by using one template and conditionally
emitting the onWrite field only when hasVectors is true. Preserve the existing
emitted key order and all other option values, including authField and
globalDatabaseField, so package-sort and API snapshots remain unchanged.

Source: Path instructions


4429-4435: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚖️ Poor tradeoff

Optional: emit one writer factory instead of seven identical createShardCtxDb blocks.

All seven admin/migration/CDC overrides now emit the same five-option writer with the same broadcast body; this PR had to touch each one. A single emitted private helper (e.g. private adminWriter(): DatabaseWriterLike) would make the next option addition a one-line change.

Also applies to: 4880-4888, 4923-4931, 4973-4981, 4994-5002, 5022-5030, 5053-5061

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/codegen/src/emit.ts` around lines 4429 - 4435, Consolidate the seven
repeated createShardCtxDb writer overrides in the affected admin, migration, and
CDC emission paths into one emitted private helper, such as adminWriter(),
returning the shared DatabaseWriterLike configuration and broadcast behavior.
Replace each duplicate five-option writer block with calls to that helper while
preserving the existing per-context behavior.
packages/shard-engine/src/read-footprint.ts (1)

61-67: 📐 Maintainability & Code Quality | 🔵 Trivial | 💤 Low value

ranges() hands back the live internal map.

The returned Map is the same object onReadRange keeps appending to, and consumers retain it long-term (SubscriptionMemo.ranges in packages/do/src/shard-do.ts). Any read after ranges() on the same footprint would mutate a memo in place. Single-use per run today, so this is a contract to pin rather than a live bug — either document "call once, then discard" on the interface or return a copy.

♻️ Document the single-use contract
     /**
      * Slices per table, for tables read ONLY through ranges. `undefined` when
      * nothing was narrowable, which lets callers skip the map entirely.
+     *
+     * Returns the tracker's OWN map, so it must be read once the read has
+     * finished; a later `onReadRange` would mutate a map the caller retained.
      */
     ranges: () => Map<string, KeyRange[]> | undefined;
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/read-footprint.ts` around lines 61 - 67, Document
the single-use contract for the ranges() method: callers must invoke it once
after all reads are recorded and discard the footprint afterward, because the
returned Map is the live internal state. Update the relevant interface or method
documentation near ranges() without changing its implementation.
packages/shard-engine/src/reactive-cache.ts (2)

355-364: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

The comment promises a snapshot that isn't taken.

dropRangeDeps iterates byRange directly while dropEntry deletes from it. That happens to be safe — a Map iterator tolerates deletion of the current entry, and an entry removed before it is visited no longer needs visiting — but the "Snapshot the pairs" comment describes code that doesn't exist and invites someone to "restore" a copy that would then re-visit dead ranges. Reword to state why in-place iteration is safe.

♻️ Say what the loop actually relies on
-        // Snapshot the pairs: dropping an entry mutates `rangeIndex` underneath.
+        // Iterated in place: `dropEntry` mutates `byRange` underneath, which a
+        // Map iterator tolerates — a bucket emptied and removed before we reach
+        // it has no entries left to drop, so skipping it is correct.
         for (const [range, keys] of byRange) {
🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/reactive-cache.ts` around lines 355 - 364, Reword
the comment immediately above the byRange iteration to describe that the loop
intentionally iterates the Map in place while dropEntry may delete entries,
relying on Map’s deletion-safe iterator behavior and skipping ranges removed
before visitation. Remove the inaccurate “Snapshot the pairs” wording without
changing the iteration or invalidation logic.

104-111: 🚀 Performance & Scalability | 🔵 Trivial | ⚖️ Poor tradeoff

Identity-keyed ranges mean buckets are never actually shared.

buildIndexRange mints a fresh KeyRange per read, so two cached queries over the identical slice land in two rangeIndex buckets — the "range shared by several cached queries" case at Line 422 can't occur in practice. Correctness holds (each bucket is still tested), but dropRangeDeps then runs keysTouchRanges once per cache entry per write instead of once per distinct slice, and keysTouchRanges allocates a filter array on each call. Keying by an encoded table|index|lo|hi string would collapse duplicates and make the shared-bucket comment true.

Also applies to: 208-224, 422-423

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/reactive-cache.ts` around lines 104 - 111, The
rangeIndex currently keys buckets by fresh KeyRange object identity, preventing
identical slices from sharing buckets. Update buildIndexRange and the rangeIndex
usage in dropRangeDeps and the shared-range handling around the affected
dependency cleanup logic to derive a stable encoded key from table, index, lo,
and hi, while retaining the structured KeyRange values for range comparisons and
preserving existing cleanup behavior.
packages/shard-engine/src/ctx-db-companions.ts (1)

665-667: 🚀 Performance & Scalability | 🔵 Trivial | 🏗️ Heavy lift

Avoid deriving index keys twice per insert.

The ctx-db.ts wiring derives rowIndexKeys(table, document) for invalidateCache, while Line 667 derives the same keys again for broadcast. Compute the keys once and pass the shared value through both contracts; this matters more for bulk inserts.

As per path instructions, packages/**/*.ts code should follow DRY principles and check performance optimizations.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/ctx-db-companions.ts` around lines 665 - 667,
Update the insert flow around invalidateCache and broadcast to compute index
keys once, store the result, and reuse it for both contracts, including the
ctx-db.ts rowIndexKeys wiring. Preserve the existing insert behavior while
eliminating duplicate derivation for bulk inserts.

Source: Path instructions

packages/sql-store/src/point-read-batcher.ts (1)

63-137: 📐 Maintainability & Code Quality | 🔵 Trivial | ⚡ Quick win

Consider adding unit tests for the coalescer.

This is new, non-trivial concurrency-adjacent logic (same-tick dedup, chunk-boundary behavior at maxBatch, per-chunk failure isolation, missing-row-as-undefined). No test file is included for it in this changeset; targeted tests (fake fetchMany, assert one call per tick/table, assert chunk cap, assert isolated chunk failure) would be cheap to write given the function has no external DB dependency.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/sql-store/src/point-read-batcher.ts` around lines 63 - 137, Add
targeted unit tests for createPointReadBatcher using a fake fetchMany
implementation: verify same-tick duplicate loads share one fetch per table and
resolve all waiters, requests are split at maxBatch, a failed chunk rejects only
its own waiters while later chunks continue, and missing rows resolve as
undefined.
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/codegen/src/emit.ts`:
- Around line 3744-3745: Remove IndexKeyEntry from the emitted shard signature
imports while retaining KeyRange. Update the import emission logic in emit.ts so
generated shards and fixtures no longer include the unused type, without
changing other generated signatures.

In `@packages/do/src/shard-do.ts`:
- Around line 3225-3236: Update transactionHeadroom() and the subscription
refresh call path to stop using the shared refreshingSubscriptions flag to
select a tracker. Create and pass a dedicated TransactionHeadroomTracker
explicitly through the refresh operation, while normal concurrent RPC dispatches
continue using currentTransactionHeadroom; preserve the existing per-refresh
bounded budget behavior.

In `@packages/shard-engine/src/ctx-db.ts`:
- Around line 2833-2839: Global insert broadcasts compute indexKeys from the
unstored caller document, causing incorrect slice invalidation decisions. In
packages/shard-engine/src/ctx-db.ts lines 2833-2839 and 2932-2938, remove
indexKeys from the single-insert and insertManyUnsafe global broadcast payloads
so the invalidation gate falls back to whole-table invalidation; do not change
the stored-row writes.
- Around line 1585-1591: The read meter misses rows materialized by scan-shaped
reads. In packages/shard-engine/src/ctx-db.ts lines 1585-1591, update the
object-form full-scan flow around onRead to charge headroom for the decoded
document count after the decode loop, since the single *scan dependency bypasses
per-row stamps; in packages/shard-engine/src/ctx-db.ts lines 1177-1198, update
meterRows usage to receive the pre-filter row count so terminal filter
operations charge the full scan window rather than only surviving rows.
- Around line 2971-2977: Update insertManyUnsafe so each row is charged to
headroom only in the pre-write recordWrite loop. When notifying through the
post-write onWrite path, suppress its wrapper’s metering for these
already-charged rows while preserving the notification behavior.

In `@packages/shard-engine/src/estimate-bytes.ts`:
- Around line 18-20: Update the byte-estimation documentation around the
JSON.stringify length calculation to explicitly state that String.length
measures UTF-16 code units rather than UTF-8 bytes, and that non-ASCII content
may be underestimated. Keep the existing estimation behavior unchanged and
clarify the approximation where the maxWrittenBytes cap is documented.

In `@packages/shard-engine/src/reactive-cache.ts`:
- Around line 184-188: Update the estimateBytes call in the cache entry
construction to use the instance-configured this.maxBytes as the fallback charge
instead of DEFAULT_MAX_BYTES, matching the limit enforced by evict().

In `@packages/sql-store/src/point-read-batcher.ts`:
- Around line 39-42: Validate PointReadBatcherOptions.maxBatch before using it
as the offset increment in the batching loop, rejecting or replacing zero and
negative values with a safe positive batch size while preserving the documented
default behavior. Update the PointReadBatcher construction and loop around
maxBatch so every iteration advances and cannot hang.

---

Nitpick comments:
In `@packages/codegen/src/emit.ts`:
- Around line 4286-4317: Deduplicate the two databaseOptions template branches
by using one template and conditionally emitting the onWrite field only when
hasVectors is true. Preserve the existing emitted key order and all other option
values, including authField and globalDatabaseField, so package-sort and API
snapshots remain unchanged.
- Around line 4429-4435: Consolidate the seven repeated createShardCtxDb writer
overrides in the affected admin, migration, and CDC emission paths into one
emitted private helper, such as adminWriter(), returning the shared
DatabaseWriterLike configuration and broadcast behavior. Replace each duplicate
five-option writer block with calls to that helper while preserving the existing
per-context behavior.

In `@packages/shard-engine/src/ctx-db-companions.ts`:
- Around line 665-667: Update the insert flow around invalidateCache and
broadcast to compute index keys once, store the result, and reuse it for both
contracts, including the ctx-db.ts rowIndexKeys wiring. Preserve the existing
insert behavior while eliminating duplicate derivation for bulk inserts.

In `@packages/shard-engine/src/index-key-codec.ts`:
- Around line 93-94: Hoist a single TextEncoder instance to module scope and
update encodeString to reuse it instead of constructing a new encoder on each
call. Preserve the existing string-to-byte encoding behavior.

In `@packages/shard-engine/src/reactive-cache.ts`:
- Around line 355-364: Reword the comment immediately above the byRange
iteration to describe that the loop intentionally iterates the Map in place
while dropEntry may delete entries, relying on Map’s deletion-safe iterator
behavior and skipping ranges removed before visitation. Remove the inaccurate
“Snapshot the pairs” wording without changing the iteration or invalidation
logic.
- Around line 104-111: The rangeIndex currently keys buckets by fresh KeyRange
object identity, preventing identical slices from sharing buckets. Update
buildIndexRange and the rangeIndex usage in dropRangeDeps and the shared-range
handling around the affected dependency cleanup logic to derive a stable encoded
key from table, index, lo, and hi, while retaining the structured KeyRange
values for range comparisons and preserving existing cleanup behavior.

In `@packages/shard-engine/src/read-footprint.ts`:
- Around line 61-67: Document the single-use contract for the ranges() method:
callers must invoke it once after all reads are recorded and discard the
footprint afterward, because the returned Map is the live internal state. Update
the relevant interface or method documentation near ranges() without changing
its implementation.

In `@packages/shard-engine/src/read-write-set.ts`:
- Around line 295-313: Refactor keysTouchRanges to build a single Map keyed by
index name from keys before iterating ranges, then retrieve each range’s
matching entries from that Map instead of calling keys.filter per range.
Preserve the existing behavior for missing indexes and rangeContains checks.

In `@packages/shard-engine/src/shard-ring.ts`:
- Around line 78-88: Move the TextEncoder allocation out of fnv1a64 and create
one module-level encoder instance for reuse. Update fnv1a64 to encode input
through that shared instance while preserving the existing hashing behavior.
- Around line 129-150: Extract the duplicated positive-integer validation from
bootstrapDirectory and vnodeForId into a shared assertPositiveInteger(value,
name) helper. Have the helper throw the consistent RangeError message and invoke
it for vnodeCount in both functions, preserving the existing validation
behavior.

In `@packages/sql-store/src/point-read-batcher.ts`:
- Around line 63-137: Add targeted unit tests for createPointReadBatcher using a
fake fetchMany implementation: verify same-tick duplicate loads share one fetch
per table and resolve all waiters, requests are split at maxBatch, a failed
chunk rejects only its own waiters while later chunks continue, and missing rows
resolve as undefined.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 21589fc9-1335-49ec-9b69-dbab48404d4b

📥 Commits

Reviewing files that changed from the base of the PR and between 80ea0ce and 20dbd94.

⛔ Files ignored due to path filters (21)
  • api-snapshots/do.api.md is excluded by none and included by none
  • api-snapshots/errors.api.md is excluded by none and included by none
  • api-snapshots/lunora.api.md is excluded by none and included by none
  • api-snapshots/shard-engine.api.md is excluded by none and included by none
  • examples/auth-playground/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/blog/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/expo/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/notify-demo/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/offline-rejections/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/payment-demo/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/realtime-cursors/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/todo-app/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • packages/codegen/__tests__/fixtures/simple/expected/_generated/shard.ts is excluded by !**/_generated/**, !**/__tests__/** and included by packages/**
  • packages/shard-engine/__tests__/ctx-db.headroom.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/index-key-codec.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/reactive-cache.ranges.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/read-write-set.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/shard-ring.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/subscription-range-gate.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/transaction-headroom.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/sql-store/__tests__/point-read-batcher.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (18)
  • packages/codegen/src/emit.ts
  • packages/do/src/index.ts
  • packages/do/src/shard-do.ts
  • packages/errors/src/catalog.ts
  • packages/shard-engine/src/ctx-db-companions.ts
  • packages/shard-engine/src/ctx-db.ts
  • packages/shard-engine/src/estimate-bytes.ts
  • packages/shard-engine/src/index-key-codec.ts
  • packages/shard-engine/src/index.ts
  • packages/shard-engine/src/reactive-cache.ts
  • packages/shard-engine/src/read-footprint.ts
  • packages/shard-engine/src/read-write-set.ts
  • packages/shard-engine/src/schema-types.ts
  • packages/shard-engine/src/shard-ring.ts
  • packages/shard-engine/src/subscription-range-gate.ts
  • packages/shard-engine/src/transaction-headroom.ts
  • packages/sql-store/src/ctx-db.ts
  • packages/sql-store/src/point-read-batcher.ts

Comment thread packages/codegen/src/emit.ts Resolved
Comment thread packages/do/src/shard-do.ts Resolved
Comment thread packages/shard-engine/src/ctx-db.ts Resolved
Comment thread packages/shard-engine/src/ctx-db.ts Resolved
Comment thread packages/shard-engine/src/ctx-db.ts Resolved
@codspeed

codspeed Bot commented Jul 30, 2026 •

Copy link
Copy Markdown

Merging this PR will improve performance by 12.15%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

⚡ 1 improved benchmark
✅ 250 untouched benchmarks
⏩ 10 skipped benchmarks1

Performance Changes

Benchmark BASE HEAD Efficiency
⚡ count, no attributes 64.8 µs 57.8 µs +12.15%

Tip

Curious why this is faster? Comment @codspeedbot explain why this is faster on this PR, or directly use the CodSpeed MCP with your agent.


Comparing feat/range-invalidation (7e84ede) with alpha (7353ddf)2

Open in CodSpeed

Footnotes

  1. 10 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on alpha (37afb05) during the generation of this report, so 7353ddf was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

prisis added a commit that referenced this pull request Jul 31, 2026
Review of #239 surfaced three correctness bugs in the new code plus a CI
break; all are fixed here.

Generated shards imported `IndexKeyEntry` but only ever inferred it, so every
downstream project failed `noUnusedLocals` — this was the `Lint (types)`
failure. Dropped from the emitted imports; `KeyRange` stays because generated
signatures name it.

A deferred subscription re-run was selected by a `refreshingSubscriptions`
instance flag, which says "a refresh is in flight", not "this caller is one".
The drain runs in the background, so a concurrent `/rpc` dispatch building its
`ctx.db` during it also took that branch and got a throwaway meter — losing
its own ceiling entirely. The re-run's budget is now threaded by value from
the generated `executeSubscription`, the way identity already is.

`insertManyUnsafe` charged every row twice: once in the pre-write loop that
bounds the batch, then again through the metering `onWrite` wrapper. It now
notifies through the raw hook, so the effective ceiling is what it says.

`.global()` insert broadcasts derived index keys from an image that was never
stored — defaults and `_creationTime` are applied on the far side, so a key
could name a position the row does not occupy and prove a write outside a
slice that contains it. Those broadcasts now omit keys and fall back to
whole-table invalidation.

Also: the read meter could not see scan-shaped reads (a full scan stamps one
`*scan` dep, and a `.filter()` terminal returned only survivors) — both now
charge the window they materialized; a non-positive `maxBatch` spun the
batcher's chunk loop forever; the cache's unserializable-entry fallback
charged the module default rather than the instance cap; `estimateBytes`
documents that it measures UTF-16 code units, not bytes; and the ring's
64-bit truncation uses `BigInt.asUintN` so the wrap-around is explicit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@prisis

prisis commented Jul 31, 2026

Copy link
Copy Markdown
Member Author

All 11 review comments are fixed and resolved; CI is 38 pass / 0 fail apart from CodSpeed. On that last one:

The CodSpeed regression looks like measurement noise, not a real change. Evidence:

  1. The flagged set changed completely between runs, and the headline moved 26.19% → 14.7%.

    • Run 1: rls-overhead :: baseline: no RLS in the chain, normalize-log-fields :: flat 3 primitives
    • Run 2: rank-page-hydrate :: in-batch, rpc-dispatch :: + resolveIdentity
      No benchmark was flagged twice.
  2. None of the four imports code this PR changes.

    • rls-overhead runs against a FakeDatabase — it never reaches ctx-db.
    • rank-page-hydrate imports only createSqliteExec; no createShardCtxDb, so it never enters the metered read path.
    • rpc-dispatch is in @lunora/runtime — this PR changes 0 files there (git diff --name-only origin/alpha | grep packages/runtime/ is empty).
    • normalize-log-fields is log-field normalisation, untouched here.
  3. Local variance covers the reported delta. Two back-to-back runs of rank-page-hydrate on this branch: 14,107 vs 14,017 ops/s, with max latency swinging 1.41 ms vs 0.96 ms — a ~47% tail swing between identical runs.

  4. CodSpeed itself reports "Different runtime environments detected — some benchmarks with significant performance changes were compared across different runtime environments, which may affect the accuracy of the results."

The read path did gain real work (a wrapper on onRead, a meterRows call per terminal), so a genuine small cost there would be legitimate — but none of the flagged benchmarks measure it. Happy to add a benchmark that actually exercises the metered read path if that would be more useful than acknowledging these.

Leaving the acknowledgement to a maintainer rather than clicking it myself.

prisis added a commit that referenced this pull request Jul 31, 2026
Review of #239 surfaced three correctness bugs in the new code plus a CI
break; all are fixed here.

Generated shards imported `IndexKeyEntry` but only ever inferred it, so every
downstream project failed `noUnusedLocals` — this was the `Lint (types)`
failure. Dropped from the emitted imports; `KeyRange` stays because generated
signatures name it.

A deferred subscription re-run was selected by a `refreshingSubscriptions`
instance flag, which says "a refresh is in flight", not "this caller is one".
The drain runs in the background, so a concurrent `/rpc` dispatch building its
`ctx.db` during it also took that branch and got a throwaway meter — losing
its own ceiling entirely. The re-run's budget is now threaded by value from
the generated `executeSubscription`, the way identity already is.

`insertManyUnsafe` charged every row twice: once in the pre-write loop that
bounds the batch, then again through the metering `onWrite` wrapper. It now
notifies through the raw hook, so the effective ceiling is what it says.

`.global()` insert broadcasts derived index keys from an image that was never
stored — defaults and `_creationTime` are applied on the far side, so a key
could name a position the row does not occupy and prove a write outside a
slice that contains it. Those broadcasts now omit keys and fall back to
whole-table invalidation.

Also: the read meter could not see scan-shaped reads (a full scan stamps one
`*scan` dep, and a `.filter()` terminal returned only survivors) — both now
charge the window they materialized; a non-positive `maxBatch` spun the
batcher's chunk loop forever; the cache's unserializable-entry fallback
charged the module default rather than the instance cap; `estimateBytes`
documents that it measures UTF-16 code units, not bytes; and the ring's
64-bit truncation uses `BigInt.asUintN` so the wrap-around is explicit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@prisis
prisis force-pushed the feat/range-invalidation branch from e12ed79 to 04d5152 Compare July 31, 2026 07:20
@prisis
prisis force-pushed the feat/range-invalidation branch from 04d5152 to 38db964 Compare July 31, 2026 07:21
Comment thread packages/shard-engine/src/shard-ring.ts Resolved
prisis and others added 4 commits July 31, 2026 09:26
`ctx.db.get` against a `.global()` backend is a network round-trip, so the
idiomatic join — `Promise.all(ids.map(ctx.db.get))` — paid one hop per id.
`Promise.all` starts every read in the same tick, so they can be collected
before the microtask queue drains and answered with a single `IN (...)`.

Batches are per (table, tick) and capped at 50 ids, matching the chunk size
the relation pre-resolver already uses against D1's parameter limit. Covers
D1 and Hyperdrive alike, since both `.global()` backends route through
`@lunora/sql-store`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Routing groundwork for progressive sharding: a fixed vnode ring, FNV-1a and
jump consistent hashing, and a directory that expresses "everything is still
local". Placement is a pure function of the document id, so a client never
names a shard.

Bootstraps at `shardCount: 0`, where every vnode maps to LOCAL_SHARD and
placement is byte-identical to today's single-DO topology. Rebalance, the
per-shard WAL and watermark protocol, and autoscale are deliberately absent —
each needs its own design and a half-built version risks misrouting live rows.

Nothing consumes this yet, so it stays off the package barrel. Named for
vnodes rather than shards because `@lunora/platform` already owns a
`ShardDirectory`/`resolveShard` pair for a different concept.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
A Durable Object is a single-threaded process with a hard memory ceiling, so
one unbounded mutation takes the whole shard down and the only diagnostic is
an isolate that vanished. That is a bad way to learn a query needed an index.

Reads and writes are now charged against per-transaction ceilings (32 MiB
written, 50k rows written, 100k rows read) and the transaction stops with a
`TRANSACTION_LIMIT_EXCEEDED` naming the ceiling it hit. Metering rides on the
`onRead`/`onWrite` hooks every path already awaits, so no write bypasses it;
`insertManyUnsafe` charges its batch before the multi-row INSERT, since
metering after the fact would let the batch materialize in full first.

The counters deliberately have no rollback path — the bytes were materialized
whether or not a savepoint kept them, and a meter that rewound could not bound
a retry loop.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`ctx.db.query(table)` stamped its dependency eagerly, before the chain
revealed whether `.withIndex(...)` was coming, so it had to assume the whole
table — and every live query on a table re-ran on every write to it. The
dependency now comes from the reader's terminal, where the plan is final, and
an indexed read records the contiguous index slice it actually touched.

Slices are half-open `[lo, hi)` over an order-preserving encoding that matches
SQLite's ordering, including UTF-8 byte order for text (JS compares UTF-16
code units, which disagrees on astral characters). Half-open is what makes
exclusive bounds correct on compound indexes.

Ranges travel as objects on their own channel, never encoded into the
row-dependency string space: a document id is arbitrary user data, so a range
packed into the same strings could be forged by an id shaped like one — and a
forged range reads as "this table is narrowed", which suppresses invalidations
rather than adding them.

Write positions are computed once on the write path, unioned across the row's
before- and after-images, and carried on the delta. A consumer working from
the post-image alone would see only where a patched row landed, never the
slice it left, and a subscriber watching that slice would never be woken.

Subscriptions report their own read footprint rather than reading a shared
per-request field: a deferred re-run interleaves with unrelated dispatches, so
a shared field would stamp one request's slices onto another subscriber's memo.
A re-run also gets its own resource budget instead of spending the writing
mutation's.

Narrowing is an optimization, never a correctness input — an unencodable
value, an unmodelled comparator, an unknown write position, or a table read
in more than one way all fall back to whole-table invalidation.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
prisis and others added 4 commits July 31, 2026 09:26
Review of #239 surfaced three correctness bugs in the new code plus a CI
break; all are fixed here.

Generated shards imported `IndexKeyEntry` but only ever inferred it, so every
downstream project failed `noUnusedLocals` — this was the `Lint (types)`
failure. Dropped from the emitted imports; `KeyRange` stays because generated
signatures name it.

A deferred subscription re-run was selected by a `refreshingSubscriptions`
instance flag, which says "a refresh is in flight", not "this caller is one".
The drain runs in the background, so a concurrent `/rpc` dispatch building its
`ctx.db` during it also took that branch and got a throwaway meter — losing
its own ceiling entirely. The re-run's budget is now threaded by value from
the generated `executeSubscription`, the way identity already is.

`insertManyUnsafe` charged every row twice: once in the pre-write loop that
bounds the batch, then again through the metering `onWrite` wrapper. It now
notifies through the raw hook, so the effective ceiling is what it says.

`.global()` insert broadcasts derived index keys from an image that was never
stored — defaults and `_creationTime` are applied on the far side, so a key
could name a position the row does not occupy and prove a write outside a
slice that contains it. Those broadcasts now omit keys and fall back to
whole-table invalidation.

Also: the read meter could not see scan-shaped reads (a full scan stamps one
`*scan` dep, and a `.filter()` terminal returned only survivors) — both now
charge the window they materialized; a non-positive `maxBatch` spun the
batcher's chunk loop forever; the cache's unserializable-entry fallback
charged the module default rather than the instance cap; `estimateBytes`
documents that it measures UTF-16 code units, not bytes; and the ring's
64-bit truncation uses `BigInt.asUintN` so the wrap-around is explicit.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`insertManyUnsafe` is optional on `DatabaseWriterLike` — the global/D1 twin has
no batch primitive — so calling it directly failed `tsc` under
`strictNullChecks`. Narrow it explicitly instead.

This was the remaining `Lint (types)` CI failure. It went unnoticed locally
because the per-package check was being filtered to `src/` errors, which hid
everything under `__tests__/`.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
`vitest/no-conditional-in-test` (warning, and warnings are errors here) rejects
the narrowing guard inside the test body. Move it into a module-level helper
so the optional `insertManyUnsafe` is narrowed once, outside any test.

Caught by the repo-wide `lint:eslint`, which I had not re-run after the
previous fix — the per-package check I was using ran before that edit existed.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The static analyzer kept flagging the 64-bit truncation as an implicit
undefined-to-number conversion, even after moving to `BigInt.asUintN`. The
arithmetic was already correct — `tsc` is clean and the placement tests pin
the hash values — but folding a mix of masks, shifts and multiplies into
compound assignments leaves the operand widths to be inferred, by a reader as
much as by a tool.

Each step now binds to an explicitly-typed intermediate, and the LCG
multiplier is a named constant rather than a literal buried in the loop.
Behaviour is unchanged.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@prisis
prisis force-pushed the feat/range-invalidation branch from 75cb9a3 to 9828d8d Compare July 31, 2026 07:33
`assertRequiredPackages` (added on alpha in #240) reads the project manifest
and treats one that EXISTS as authoritative — "declares nothing" — while a
missing manifest means "cannot tell" and skips the check. This test writes a
bare `{ "packageManager": "npm@10.9.0" }` purely to exercise
`detectPackageManager`, so the gate started reading it as a project declaring
no dependencies and failed the deploy on the fixture's `.global()` tables.

The test's intent is "the project declares npm", not "the project declares
nothing", so the manifest now carries `@lunora/d1` alongside `packageManager`.

Pre-existing on alpha — reproduced on a clean checkout of 37afb05 with none of
this branch's changes, and fixed there before porting. Unrelated to the rest of
this PR; split out so it can be cherry-picked.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
The test dialect hard-coded `supportsFts5: true`, but FTS5 is a property of
the RUNTIME, not the dialect: Node 24 bundles a SQLite with the module, Node
22 does not. The suite therefore passed on one CI matrix leg and failed the
other with `no such module: fts5`.

Probe once and report the real answer. Both legs stay meaningful rather than
one being skipped — where FTS5 exists the suite exercises the `fts5` search
layout, and where it does not it exercises the portable `inverted` layout,
which is the split the store already models for backends without FTS5.
Verified passing with the probe forced both ways.

Latent since the search work landed: these tests only run when `sql-store` is
affected, so the matrix split went unnoticed until a change touched it.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 3

🧹 Nitpick comments (2)
packages/shard-engine/src/ctx-db.ts (2)

1260-1278: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

Lazy iteration stamps one identical range per page.

Every reader.paginate call runs stampTerminal() (Line 1308). createReadFootprint.onReadRange appends each reported range to byTable without deduplication (read-footprint.ts Lines 50-60). A for await walk over a large table therefore stores ceil(rows / 128) copies of the same KeyRange in the subscription memo, and keysTouchRanges rescans all of them for every changed index key.

Deduplicate in createReadFootprint.onReadRange — compare index, lo, and hi before pushing — so the memo size stays bound to the distinct slices a query read.

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/ctx-db.ts` around lines 1260 - 1278, Update
createReadFootprint.onReadRange to avoid appending duplicate ranges to byTable:
before pushing, compare the incoming range’s index, lo, and hi against existing
entries and retain it only when no identical range exists. Keep distinct ranges
unchanged so lazy pagination no longer grows the subscription memo with repeated
copies.

2400-2403: 🚀 Performance & Scalability | 🔵 Trivial | ⚡ Quick win

rowIndexKeys is recomputed for every consumer on each write path. Each write site calls rowIndexKeys once for the cache invalidation and again for the broadcast with identical arguments. Every call re-encodes each declared index key for each supplied image, so the encode cost is paid twice per write.

  • packages/shard-engine/src/ctx-db.ts#L2400-L2403: bind rowIndexKeys(tableName, existing, merged) to a local in the soft-delete branch and pass it to cache?.invalidate and broadcast.
  • packages/shard-engine/src/ctx-db.ts#L1999-L2000: compute the entries once inside createCompanionSync and pass them to both invalidateCache and indexKeysFor, since syncCompanionsForInsert calls both for the same image.
  • packages/shard-engine/src/ctx-db.ts#L2437-L2440: bind rowIndexKeys(tableName, existing) to a local in the physical-delete branch and reuse it.
  • packages/shard-engine/src/ctx-db.ts#L3208-L3215: bind rowIndexKeys(tableName, existing, merged) to a local in patch and reuse it.
  • packages/shard-engine/src/ctx-db.ts#L3629-L3632: bind rowIndexKeys(tableName, previous, replaced) to a local in replace and reuse it.

As per path instructions, "The code adheres to best practices recommended for performance."

🤖 Prompt for AI Agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

In `@packages/shard-engine/src/ctx-db.ts` around lines 2400 - 2403, Avoid
duplicate rowIndexKeys computation by binding each result once and reusing it
across consumers: in packages/shard-engine/src/ctx-db.ts lines 2400-2403
(soft-delete), 2437-2440 (physical-delete), 3208-3215 (patch), and 3629-3632
(replace), pass the local to cache invalidation and broadcast paths; in lines
1999-2000, compute entries once in createCompanionSync and reuse them for
invalidateCache and indexKeysFor.

Source: Path instructions

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/shard-engine/src/ctx-db.ts`:
- Around line 2933-2938: Global insert paths bypass write-meter accounting. In
packages/shard-engine/src/ctx-db.ts lines 2933-2938, update the single-insert
global branch to call headroom.recordWrite with the inserted row before
broadcast; in lines 3031-3036, call headroom.recordWrite for each row inside the
insertManyUnsafe global loop, matching the local pre-charge behavior.
- Around line 1322-1326: Update the pagination path around paginateStage and
meterRows to charge the pre-filter scan count, matching runFetch, rather than
only page.page.length. Use the scanned-row count exposed by paginateStage and
report whichever is larger when unbounded filtering materializes more rows than
the returned page.
- Around line 1216-1225: The geo fetch path currently meters only the sliced
result because runFetch receives no pre-slice count from runGeoTerminal. Update
runGeoTerminal and its call in runFetch to propagate the number of matched
documents before applying the limit, assign that value to scanned, and keep the
existing meterRows(Math.max(scanned, rows.length)) behavior so limited geo reads
report the full docs.length.

---

Nitpick comments:
In `@packages/shard-engine/src/ctx-db.ts`:
- Around line 1260-1278: Update createReadFootprint.onReadRange to avoid
appending duplicate ranges to byTable: before pushing, compare the incoming
range’s index, lo, and hi against existing entries and retain it only when no
identical range exists. Keep distinct ranges unchanged so lazy pagination no
longer grows the subscription memo with repeated copies.
- Around line 2400-2403: Avoid duplicate rowIndexKeys computation by binding
each result once and reusing it across consumers: in
packages/shard-engine/src/ctx-db.ts lines 2400-2403 (soft-delete), 2437-2440
(physical-delete), 3208-3215 (patch), and 3629-3632 (replace), pass the local to
cache invalidation and broadcast paths; in lines 1999-2000, compute entries once
in createCompanionSync and reuse them for invalidateCache and indexKeysFor.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 2db75fbb-2ed5-4b8d-a1c4-98d1f838cc2c

📥 Commits

Reviewing files that changed from the base of the PR and between ece1076 and 1a81e56.

⛔ Files ignored due to path filters (23)
  • api-snapshots/do.api.md is excluded by none and included by none
  • api-snapshots/errors.api.md is excluded by none and included by none
  • api-snapshots/lunora.api.md is excluded by none and included by none
  • api-snapshots/shard-engine.api.md is excluded by none and included by none
  • examples/auth-playground/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/blog/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/expo/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/notify-demo/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/offline-rejections/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/payment-demo/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/realtime-cursors/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • examples/todo-app/lunora/_generated/shard.ts is excluded by !**/_generated/** and included by none
  • packages/cli/__tests__/commands/deploy.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/codegen/__tests__/fixtures/simple/expected/_generated/shard.ts is excluded by !**/_generated/**, !**/__tests__/** and included by packages/**
  • packages/shard-engine/__tests__/ctx-db.headroom.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/index-key-codec.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/reactive-cache.ranges.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/read-write-set.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/shard-ring.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/subscription-range-gate.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/transaction-headroom.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/sql-store/__tests__/ctx-db.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/sql-store/__tests__/point-read-batcher.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (18)
  • packages/codegen/src/emit.ts
  • packages/do/src/index.ts
  • packages/do/src/shard-do.ts
  • packages/errors/src/catalog.ts
  • packages/shard-engine/src/ctx-db-companions.ts
  • packages/shard-engine/src/ctx-db.ts
  • packages/shard-engine/src/estimate-bytes.ts
  • packages/shard-engine/src/index-key-codec.ts
  • packages/shard-engine/src/index.ts
  • packages/shard-engine/src/reactive-cache.ts
  • packages/shard-engine/src/read-footprint.ts
  • packages/shard-engine/src/read-write-set.ts
  • packages/shard-engine/src/schema-types.ts
  • packages/shard-engine/src/shard-ring.ts
  • packages/shard-engine/src/subscription-range-gate.ts
  • packages/shard-engine/src/transaction-headroom.ts
  • packages/sql-store/src/ctx-db.ts
  • packages/sql-store/src/point-read-batcher.ts
🚧 Files skipped from review as they are similar to previous changes (16)
  • packages/shard-engine/src/estimate-bytes.ts
  • packages/shard-engine/src/index.ts
  • packages/shard-engine/src/schema-types.ts
  • packages/sql-store/src/ctx-db.ts
  • packages/errors/src/catalog.ts
  • packages/shard-engine/src/subscription-range-gate.ts
  • packages/shard-engine/src/transaction-headroom.ts
  • packages/shard-engine/src/index-key-codec.ts
  • packages/shard-engine/src/read-write-set.ts
  • packages/shard-engine/src/ctx-db-companions.ts
  • packages/codegen/src/emit.ts
  • packages/shard-engine/src/shard-ring.ts
  • packages/sql-store/src/point-read-batcher.ts
  • packages/shard-engine/src/reactive-cache.ts
  • packages/do/src/index.ts
  • packages/do/src/shard-do.ts

Comment thread packages/shard-engine/src/ctx-db.ts Resolved
Comment thread packages/shard-engine/src/ctx-db.ts Resolved
Comment thread packages/shard-engine/src/ctx-db.ts Resolved
Three paths still slipped the transaction ceilings, all in the same family as
the earlier fixes — the meter was attached to the paths that report rows, not
to every path that materializes them.

Geo reads: `runGeoFetch` decodes and scores every candidate in the covering
geohash set, then slices to the limit, so `.take(1)` over a wide radius could
materialize thousands of rows while the meter saw one. It now reports the
pre-slice candidate count.

Filtered pagination: `paginateStage` skips the SQL `LIMIT` when in-memory
predicates are staged, so filling a one-row page can scan the whole table. It
now reports the pre-filter row count, and `paginate` charges whichever of scan
or page is larger.

`.global()` inserts: both global branches return before `onWrite`, which is
where the meter charges — so a transaction could write unbounded rows to a
global table without consuming `maxWrittenRows` or `maxWrittenBytes` at all.
Both branches now charge directly.

Adds a regression test for the pagination case, which fails under the old
page-length accounting.

Co-Authored-By: Claude Opus 5 <noreply@anthropic.com>
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — 7e84ede7 Deployed Jul 31, 2026 by prisis via Benchmarks #1260
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant