Skip to content

feat(shard-engine): surface FTS score and geo distance via collectWithScores - #267

Merged
prisis merged 3 commits into
alphafrom
advisor/236-search-geo-scores
Aug 1, 2026
Merged

prisis merged 3 commits into
alphafrom
advisor/236-search-geo-scores

Conversation

@prisis

@prisis prisis commented Aug 1, 2026 •

Copy link
Copy Markdown
Member

What

Adds an opt-in .collectWithScores() terminal to the reactive table reader that pairs each row with the ranking value the read already computes to order it — FTS relevance score for a .withSearchIndex() chain, haversine distanceMeters for a .withGeoIndex() chain — instead of computing it and throwing it away. .collect() and its bare-TDocument[] shape are untouched.

Why

Both the FTS and geo read paths compute a __score__ / distance to sort by, then discard it before returning. Callers who want the score (search-result ranking UIs, "N meters away" labels) had no way to read it back without re-querying.

Design points

  • Separate terminal, not .withScores() — keeps .collect()'s return type exactly TDocument[]; only the new terminal returns ScoredDocument[].
  • runSearchFetch is now a thin .map(e => e.document) wrapper over the scored path — the default (unscored) result shape is byte-identical to before.
  • .within() box matches carry distanceMeters: null (a box has no point-distance metric) rather than a misleading 0; .near() carries the real distance.
  • RLS/mask composition preserved: a .filter() (how RLS pushes its policy down) then .collectWithScores() returns the scored survivors. The mask wrapper masks only the document half and passes score/distanceMeters through in the clear — same reasoning that lets count/rank bypass the per-column mask.
  • .global() (@lunora/sql-store) fails closed: the three SQL search layouts order by __score__ but never select it back; surfacing it needs plumbing through all three layouts. Until then collectWithScores() throws a clear LunoraError (mirroring the existing withGeoIndex()-unsupported guard) rather than a "not a function" crash. Natural follow-up.

Verification

  • @lunora/shard-engine 852 tests, @lunora/server 484, @lunora/sql-store 91 — all green; lint:types clean on each.
  • @lunora/codegen 973 tests, no golden drift — the reader types are authored in data-model.ts, not codegen-emitted.
  • New tests: descending-score, backward-compat (bare-doc shape unchanged), RLS .filter()-then-.collectWithScores(), geo nearest-first + null-sentinel, masked-search/masked-geo (document masked, score/distance clear), and the sql-store fail-closed error.

Merge-order note

Touches packages/server/src/mask/middleware.ts and packages/shard-engine/src/ctx-db.ts. Individually mergeable vs alpha; if the mask where-guard PR (208-209) lands first, resolve middleware.ts by keeping both the guards and the new collectWithScores wrapper (disjoint methods).

Plan: plans/236-*.md.

🤖 Generated with Claude Code

Summary by CodeRabbit

  • New Features
    • Search results can now include relevance scores.
    • Geographic query results can include distances in meters.
    • Added a unified method for retrieving documents with scoring details.
    • Bounding-box geographic queries return a null distance when no specific distance applies.
  • Compatibility
    • Existing document-only collection behavior remains unchanged.
    • Unsupported scoring requests provide a clear error and guidance.

…hScores

The FTS reader and geo reader both compute a per-row ranking value
(__score__ / haversine distance), order the result set by it, then discard
it before returning bare documents — every relevance/proximity UI had to
recompute a value already produced at the sort boundary.

Adds a separate `.collectWithScores()` terminal alongside the existing
`.collect()` (design choice (a) from the plan): `{ document, score }[]` for
a `.withSearchIndex()` chain, `{ document, distanceMeters }[]` for
`.withGeoIndex()`. The default `.collect()`/`.take()`/`.paginate()` shape is
untouched — pinned by a "same documents as .collect()" test on both readers.

`.within()` box matches have no point-distance metric, so `distanceMeters`
is `null` for those rows rather than the misleading `0` the internal scorer
used to carry. `.near()` rows keep their real haversine distance.

The new terminal goes through the exact same policy path `.collect()` does:
- RLS composes for free — it forwards the reader object by reference and
  pushes its row policy as `.filter()` on the shared query stage, which
  `.collectWithScores()` reads from identically. Pinned with a
  `.filter()`-then-`.collectWithScores()` test on both readers.
- Masking required an explicit fix: `mask/middleware.ts`'s `wrapReader`
  reconstructs a narrow reader object per call rather than forwarding by
  reference, so it needed its own `collectWithScores` that reuses the same
  `maskRow` helper `.collect()` uses on the `document` half of each pair —
  `score`/`distanceMeters` carry no column value (same reasoning that lets
  `count`/`rank` pass the mask wrapper unmasked).

`SearchReader`/`GeoReader` in `@lunora/server`'s data-model.ts are authored
types, not codegen-emitted, so no codegen change was needed (confirmed by a
full codegen test run with no golden-fixture drift).

`@lunora/sql-store` (the `.global()` D1/PlanetScale backend) also computes
and discards the same `__score__` in its three search layouts, and shares
the `TableReaderLike` contract, so it needed a `collectWithScores` too.
Real score plumbing there touches three interchangeable storage layouts and
is follow-up work; for now it fails closed with a clear error (mirroring
the existing `withGeoIndex()` `.global()`-unsupported guard) rather than a
bare "not a function".

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
@netlify

netlify Bot commented Aug 1, 2026 •

Copy link
Copy Markdown

✅ Deploy Preview for lunorash ready!

Name Link
🔨 Latest commit daedb8d
🔍 Latest deploy log https://app.netlify.com/projects/lunorash/deploys/6a6dc8fadb2e2a0008046f79
😎 Deploy Preview https://deploy-preview-267--lunorash.netlify.app
📱 Preview on mobile
Toggle QR Code...

QR Code

Use your smartphone camera to open QR code link.
🤖 Make changes Run an agent on this branch

To edit notification comments on pull requests, go to your Netlify project configuration.

@coderabbitai

coderabbitai Bot commented Aug 1, 2026 •

Copy link
Copy Markdown

Review Change Stack

Warning

Review limit reached

@prisis, you've reached your PR review limit, so we couldn't start this review.

Next review available in: 13 minutes

Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available.
You're only billed for reviews past your plan's rate limits ($0.25/file).

How can I continue?

After more reviews become available, a review can be triggered using the @coderabbitai review command as a PR comment. Alternatively, push new commits to this PR.

To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews.

How do review limits work?

CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability.

For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window.

Please refer docs for additional details.

Review details
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 9a09194d-dac8-4f4a-80dc-d4976008d361

📥 Commits

Reviewing files that changed from the base of the PR and between 71c7a21 and daedb8d.

⛔ Files ignored due to path filters (1)
  • packages/server/__tests__/mask.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (3)
  • packages/server/src/mask/middleware.ts
  • packages/shard-engine/src/index.ts
  • packages/shard-engine/src/schema-types.ts

Walkthrough

Search and geo readers now support collectWithScores(). Search results include relevance scores. Geo results include distances or null for bounding-box queries. Masking preserves metadata. Global-table search rejects the method explicitly.

Changes

Scored query collection

Layer / File(s) Summary
Public scored-result contract
packages/shard-engine/src/schema-types.ts, packages/server/src/data-model.ts, packages/server/src/mask/middleware.ts, packages/shard-engine/src/index.ts, packages/shard-engine/src/ctx-db.ts
Adds ScoredDocument and collectWithScores() to the reader contracts and server-facing readers.
Search score execution
packages/shard-engine/src/ctx-db.ts
Carries FTS and fallback relevance scores through ranking, filtering, and terminal execution. Existing document-only collection removes scores.
Geo distance execution
packages/shard-engine/src/ctx-db.ts
Shares geo candidate resolution between bare and scored reads. Near queries return haversine distances. Within queries return null.
Reader dispatch and adapters
packages/shard-engine/src/ctx-db.ts, packages/server/src/mask/middleware.ts, packages/sql-store/src/ctx-db.ts
Adds scored dispatch and collection, masks only documents, and rejects unsupported global-table calls with an explicit error.

Estimated code review effort: 3 (Moderate) | ~25 minutes

Sequence Diagram(s)

sequenceDiagram
  participant Caller
  participant TableReaderLike
  participant ScoredFetchDispatcher
  participant SearchOrGeoExecution
  participant RowMasker
  Caller->>TableReaderLike: collectWithScores()
  TableReaderLike->>ScoredFetchDispatcher: fetch scored query
  ScoredFetchDispatcher->>SearchOrGeoExecution: execute search or geo stage
  SearchOrGeoExecution-->>ScoredFetchDispatcher: documents with score or distanceMeters
  ScoredFetchDispatcher-->>TableReaderLike: scored documents
  TableReaderLike->>RowMasker: mask documents
  RowMasker-->>Caller: masked documents with metadata preserved
Loading

Possibly related PRs

🚥 Pre-merge checks | ✅ 5
✅ Passed checks (5 passed)
Check name Status Explanation
Title check ✅ Passed The title clearly and concisely describes exposing FTS scores and geo distances through collectWithScores.
Description check ✅ Passed The description clearly covers the change, rationale, design, verification, compatibility, and merge notes, but omits the template headings and checklist details.
Docstring Coverage ✅ Passed No functions found in the changed files to evaluate docstring coverage. Skipping docstring coverage check.
Linked Issues check ✅ Passed Check skipped because no linked issues were found for this pull request.
Out of Scope Changes check ✅ Passed Check skipped because no linked issues were found for this pull request.
✨ Finishing Touches
📝 Generate docstrings
  • Create stacked PR
  • Commit on current branch
🧪 Generate unit tests (beta)
  • Create PR with unit tests
  • Commit unit tests in branch advisor/236-search-geo-scores

Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out.

❤️ Share

Comment @coderabbitai help to get the list of available commands.

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Thank you for following the naming conventions! 🙏

@github-actions

github-actions Bot commented Aug 1, 2026

Copy link
Copy Markdown
Contributor

Thank you for confirming the Contributor License Agreement! 🙏

@coderabbitai coderabbitai Bot left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Actionable comments posted: 1

🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.

Inline comments:
In `@packages/shard-engine/src/schema-types.ts`:
- Around line 362-366: Replace the shared ScoredDocument shape with an exclusive
union: search rows must require score and set distanceMeters?: never, while geo
rows must require distanceMeters (null | number) and set score?: never. Apply
the same union in packages/shard-engine/src/schema-types.ts at lines 362-366 and
packages/server/src/mask/middleware.ts at lines 84-93, preserving each file’s
existing type usage.
🪄 Autofix (Beta)

Fix all unresolved CodeRabbit comments on this PR:

  • Push a commit to this branch (recommended)
  • Create a new PR with the fixes

ℹ️ Review info
⚙️ Run configuration

Configuration used: Path: .coderabbit.yaml

Review profile: CHILL

Plan: Pro Plus

Run ID: 92a485f7-7cc1-4f90-ae54-3d8f00f258b3

📥 Commits

Reviewing files that changed from the base of the PR and between 36421ad and 71c7a21.

⛔ Files ignored due to path filters (4)
  • packages/server/__tests__/mask.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/ctx-db.geo.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/shard-engine/__tests__/ctx-db.search.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
  • packages/sql-store/__tests__/ctx-db.test.ts is excluded by !**/__tests__/**, !**/*.test.ts and included by packages/**
📒 Files selected for processing (6)
  • packages/server/src/data-model.ts
  • packages/server/src/mask/middleware.ts
  • packages/shard-engine/src/ctx-db.ts
  • packages/shard-engine/src/index.ts
  • packages/shard-engine/src/schema-types.ts
  • packages/sql-store/src/ctx-db.ts

@codspeed

codspeed Bot commented Aug 1, 2026 •

Copy link
Copy Markdown

Merging this PR will degrade performance by 13.3%

⚠️ Different runtime environments detected

Some benchmarks with significant performance changes were compared across different runtime environments,
which may affect the accuracy of the results.

Open the report in CodSpeed to investigate

❌ 2 regressed benchmarks
✅ 191 untouched benchmarks
⏩ 68 skipped benchmarks1

Warning

Please fix the performance issues or acknowledge them on CodSpeed.

Performance Changes

Benchmark BASE HEAD Efficiency
❌ in-batch: single IN(...) query + id->doc re-projection 791.6 µs 947.2 µs -16.43%
❌ flat 3 primitives (the notify.send attribute shape) 55.7 µs 61.9 µs -10.04%

Tip

Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.


Comparing advisor/236-search-geo-scores (daedb8d) with alpha (a540991)2

Open in CodSpeed

Footnotes

  1. 68 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩

  2. No successful run was found on alpha (d9791ea) during the generation of this report, so a540991 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩

collectWithScores() masked only the document half of each row,
leaving distanceMeters in the clear. withGeoIndex's .near()/.within()
builders name no column, so the existing masked-field guards never
fire for geo queries. Against a masked geoPoint column this returns
document.location === null but the exact haversine distance to the
caller's probe point — three probes trilaterate the hidden coordinate.

Null out distanceMeters on every row the mask wrapper returns from
collectWithScores(), since this module has no schema-to-geo-index-field
map to prove a given index isn't built over the masked column. score
stays in the clear: search queries name their field, so
assertIndexFieldsAllowed already throws MASK_UNSUPPORTED before a
masked-field search can run.

Add masked-geo (distanceMeters withheld), masked-search (score still
clear), and unmasked-table (real distanceMeters, wrapper never
applies) coverage to mask.test.ts.
score and distanceMeters were both optional on the same interface, so
a row could type-check with neither or both populated even though
only one is ever set per call. Split into GeoScoredDocument (required
distanceMeters, score?: never) and SearchScoredDocument (required
score, distanceMeters?: never) unioned as ScoredDocument, mirrored in
@lunora/server's mask middleware. The mask wrapper's collectWithScores
now reconstructs a row per branch instead of mutating a shared
optional field, and the test fixture emits mutually exclusive rows.

Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
Claude-Session: https://claude.ai/code/session_019ZbZz7EjzFsuuubvUyyJWN
@prisis
prisis merged commit b3ee8c2 into alpha Aug 1, 2026
40 of 44 checks passed
@prisis
prisis deleted the advisor/236-search-geo-scores branch August 1, 2026 10:57
@github-actions

github-actions Bot commented Sep 8, 2026

Copy link
Copy Markdown
Contributor

This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs.
Please note this issue tracker is not a help forum. We recommend using our GitHub Discussions tab for questions.

@github-actions github-actions Bot locked as resolved and limited conversation to collaborators Sep 8, 2026

This branch was previously deployed

1 inactive deployment
benchmarks — daedb8d1 Deployed Aug 1, 2026 by prisis via Benchmarks #1323
Sign up for free to subscribe to this conversation on GitHub. Already have an account? Sign in.

Projects

None yet

Development

Successfully merging this pull request may close these issues.

1 participant