Repository navigation
feat(shard-engine): surface FTS score and geo distance via collectWithScores - #267
Conversation
…hScores
The FTS reader and geo reader both compute a per-row ranking value
(__score__ / haversine distance), order the result set by it, then discard
it before returning bare documents — every relevance/proximity UI had to
recompute a value already produced at the sort boundary.
Adds a separate `.collectWithScores()` terminal alongside the existing
`.collect()` (design choice (a) from the plan): `{ document, score }[]` for
a `.withSearchIndex()` chain, `{ document, distanceMeters }[]` for
`.withGeoIndex()`. The default `.collect()`/`.take()`/`.paginate()` shape is
untouched — pinned by a "same documents as .collect()" test on both readers.
`.within()` box matches have no point-distance metric, so `distanceMeters`
is `null` for those rows rather than the misleading `0` the internal scorer
used to carry. `.near()` rows keep their real haversine distance.
The new terminal goes through the exact same policy path `.collect()` does:
- RLS composes for free — it forwards the reader object by reference and
pushes its row policy as `.filter()` on the shared query stage, which
`.collectWithScores()` reads from identically. Pinned with a
`.filter()`-then-`.collectWithScores()` test on both readers.
- Masking required an explicit fix: `mask/middleware.ts`'s `wrapReader`
reconstructs a narrow reader object per call rather than forwarding by
reference, so it needed its own `collectWithScores` that reuses the same
`maskRow` helper `.collect()` uses on the `document` half of each pair —
`score`/`distanceMeters` carry no column value (same reasoning that lets
`count`/`rank` pass the mask wrapper unmasked).
`SearchReader`/`GeoReader` in `@lunora/server`'s data-model.ts are authored
types, not codegen-emitted, so no codegen change was needed (confirmed by a
full codegen test run with no golden-fixture drift).
`@lunora/sql-store` (the `.global()` D1/PlanetScale backend) also computes
and discards the same `__score__` in its three search layouts, and shares
the `TableReaderLike` contract, so it needed a `collectWithScores` too.
Real score plumbing there touches three interchangeable storage layouts and
is follow-up work; for now it fails closed with a clear error (mirroring
the existing `withGeoIndex()` `.global()`-unsupported guard) rather than a
bare "not a function".
Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com>
✅ Deploy Preview for lunorash ready!
To edit notification comments on pull requests, go to your Netlify project configuration. |
|
Warning Review limit reached
Next review available in: 13 minutes Enable usage-based reviews in Billing to review now. Otherwise, wait until the next included review is available. How can I continue?After more reviews become available, a review can be triggered using the To avoid repeated limits, reduce automatic review volume by pausing incremental auto-reviews earlier, using label-based review opt-in, excluding WIP or generated PR titles, or requesting reviews manually when the PR is ready. If your team needs uninterrupted high-volume reviews, an organization admin can enable usage-based reviews. How do review limits work?CodeRabbit enforces per-developer PR review limits for each organization. Most developers receive the normal plan review availability. For paid Pro and Pro+ PR reviews, CodeRabbit uses adaptive limits for sustained high-volume activity. When a developer's recent PR review activity reaches the 95th percentile or higher among CodeRabbit users, additional reviews become available more gradually as earlier reviews age out of the rolling window. Please refer docs for additional details. Review details⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Pro Plus Run ID: ⛔ Files ignored due to path filters (1)
📒 Files selected for processing (3)
WalkthroughSearch and geo readers now support ChangesScored query collection
Estimated code review effort: 3 (Moderate) | ~25 minutes Sequence Diagram(s)sequenceDiagram
participant Caller
participant TableReaderLike
participant ScoredFetchDispatcher
participant SearchOrGeoExecution
participant RowMasker
Caller->>TableReaderLike: collectWithScores()
TableReaderLike->>ScoredFetchDispatcher: fetch scored query
ScoredFetchDispatcher->>SearchOrGeoExecution: execute search or geo stage
SearchOrGeoExecution-->>ScoredFetchDispatcher: documents with score or distanceMeters
ScoredFetchDispatcher-->>TableReaderLike: scored documents
TableReaderLike->>RowMasker: mask documents
RowMasker-->>Caller: masked documents with metadata preserved
Possibly related PRs
🚥 Pre-merge checks | ✅ 5✅ Passed checks (5 passed)
✨ Finishing Touches📝 Generate docstrings
🧪 Generate unit tests (beta)
Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
|
Thank you for following the naming conventions! 🙏 |
|
Thank you for confirming the Contributor License Agreement! 🙏 |
There was a problem hiding this comment.
Actionable comments posted: 1
🤖 Prompt for all review comments with AI agents
Verify each finding against current code. Fix only still-valid issues, skip the
rest with a brief reason, keep changes minimal, and validate.
Inline comments:
In `@packages/shard-engine/src/schema-types.ts`:
- Around line 362-366: Replace the shared ScoredDocument shape with an exclusive
union: search rows must require score and set distanceMeters?: never, while geo
rows must require distanceMeters (null | number) and set score?: never. Apply
the same union in packages/shard-engine/src/schema-types.ts at lines 362-366 and
packages/server/src/mask/middleware.ts at lines 84-93, preserving each file’s
existing type usage.
🪄 Autofix (Beta)
Fix all unresolved CodeRabbit comments on this PR:
- Push a commit to this branch (recommended)
- Create a new PR with the fixes
ℹ️ Review info
⚙️ Run configuration
Configuration used: Path: .coderabbit.yaml
Review profile: CHILL
Plan: Pro Plus
Run ID: 92a485f7-7cc1-4f90-ae54-3d8f00f258b3
⛔ Files ignored due to path filters (4)
packages/server/__tests__/mask.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/shard-engine/__tests__/ctx-db.geo.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/shard-engine/__tests__/ctx-db.search.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**packages/sql-store/__tests__/ctx-db.test.tsis excluded by!**/__tests__/**,!**/*.test.tsand included bypackages/**
📒 Files selected for processing (6)
packages/server/src/data-model.tspackages/server/src/mask/middleware.tspackages/shard-engine/src/ctx-db.tspackages/shard-engine/src/index.tspackages/shard-engine/src/schema-types.tspackages/sql-store/src/ctx-db.ts
Merging this PR will degrade performance by 13.3%
|
| Benchmark | BASE |
HEAD |
Efficiency | |
|---|---|---|---|---|
| ❌ | in-batch: single IN(...) query + id->doc re-projection |
791.6 µs | 947.2 µs | -16.43% |
| ❌ | flat 3 primitives (the notify.send attribute shape) |
55.7 µs | 61.9 µs | -10.04% |
Tip
Investigate this regression by commenting @codspeedbot fix this regression on this PR, or directly use the CodSpeed MCP with your agent.
Comparing advisor/236-search-geo-scores (daedb8d) with alpha (a540991)2
Footnotes
-
68 benchmarks were skipped, so the baseline results were used instead. If they were deleted from the codebase, click here and archive them to remove them from the performance reports. ↩
-
No successful run was found on
alpha(d9791ea) during the generation of this report, so a540991 was used instead as the comparison base. There might be some changes unrelated to this pull request in this report. ↩
collectWithScores() masked only the document half of each row, leaving distanceMeters in the clear. withGeoIndex's .near()/.within() builders name no column, so the existing masked-field guards never fire for geo queries. Against a masked geoPoint column this returns document.location === null but the exact haversine distance to the caller's probe point — three probes trilaterate the hidden coordinate. Null out distanceMeters on every row the mask wrapper returns from collectWithScores(), since this module has no schema-to-geo-index-field map to prove a given index isn't built over the masked column. score stays in the clear: search queries name their field, so assertIndexFieldsAllowed already throws MASK_UNSUPPORTED before a masked-field search can run. Add masked-geo (distanceMeters withheld), masked-search (score still clear), and unmasked-table (real distanceMeters, wrapper never applies) coverage to mask.test.ts.
score and distanceMeters were both optional on the same interface, so a row could type-check with neither or both populated even though only one is ever set per call. Split into GeoScoredDocument (required distanceMeters, score?: never) and SearchScoredDocument (required score, distanceMeters?: never) unioned as ScoredDocument, mirrored in @lunora/server's mask middleware. The mask wrapper's collectWithScores now reconstructs a row per branch instead of mutating a shared optional field, and the test fixture emits mutually exclusive rows. Co-Authored-By: Claude Opus 4.8 <noreply@anthropic.com> Claude-Session: https://claude.ai/code/session_019ZbZz7EjzFsuuubvUyyJWN
|
This pull request has been automatically locked since there has not been any recent activity after it was closed. Please open a new issue for related bugs. |
What
Adds an opt-in
.collectWithScores()terminal to the reactive table reader that pairs each row with the ranking value the read already computes to order it — FTS relevancescorefor a.withSearchIndex()chain, haversinedistanceMetersfor a.withGeoIndex()chain — instead of computing it and throwing it away..collect()and its bare-TDocument[]shape are untouched.Why
Both the FTS and geo read paths compute a
__score__/ distance to sort by, then discard it before returning. Callers who want the score (search-result ranking UIs, "N meters away" labels) had no way to read it back without re-querying.Design points
.withScores()— keeps.collect()'s return type exactlyTDocument[]; only the new terminal returnsScoredDocument[].runSearchFetchis now a thin.map(e => e.document)wrapper over the scored path — the default (unscored) result shape is byte-identical to before..within()box matches carrydistanceMeters: null(a box has no point-distance metric) rather than a misleading0;.near()carries the real distance..filter()(how RLS pushes its policy down) then.collectWithScores()returns the scored survivors. The mask wrapper masks only thedocumenthalf and passesscore/distanceMetersthrough in the clear — same reasoning that letscount/rankbypass the per-column mask..global()(@lunora/sql-store) fails closed: the three SQL search layouts order by__score__but never select it back; surfacing it needs plumbing through all three layouts. Until thencollectWithScores()throws a clearLunoraError(mirroring the existingwithGeoIndex()-unsupported guard) rather than a "not a function" crash. Natural follow-up.Verification
@lunora/shard-engine852 tests,@lunora/server484,@lunora/sql-store91 — all green;lint:typesclean on each.@lunora/codegen973 tests, no golden drift — the reader types are authored indata-model.ts, not codegen-emitted..filter()-then-.collectWithScores(), geo nearest-first + null-sentinel, masked-search/masked-geo (document masked, score/distance clear), and the sql-store fail-closed error.Merge-order note
Touches
packages/server/src/mask/middleware.tsandpackages/shard-engine/src/ctx-db.ts. Individually mergeable vsalpha; if the mask where-guard PR (208-209) lands first, resolvemiddleware.tsby keeping both the guards and the newcollectWithScoreswrapper (disjoint methods).Plan:
plans/236-*.md.🤖 Generated with Claude Code
Summary by CodeRabbit
nulldistance when no specific distance applies.