perf(core): bound temporal search context lookup - #139
Conversation
There was a problem hiding this comment.
🟡 Changes recommended
The context sorting comparator in query.ts violates the JS sort contract for equal/null timestamps, which can yield unstable or incorrect ordering in the new temporal-context scenarios.
Once you've addressed the issues Copilot identified, you can request another Copilot review.
Pull request overview
This PR optimizes createQueryApi().search() temporal-context lookup by avoiding per-hit full-session scans and instead bounding candidate selection via the existing (session_id, timestamp) index while preserving the prior distance/tie ordering behavior for canonical timestamps.
Changes:
- Prepare and reuse two temporal-context statements per
search()call (indexed path for canonical timestamps; scan fallback otherwise). - Bound temporal neighbor candidates to a small window (plus limited null-timestamp rows) before applying
JULIANDAY()distance ordering. - Add a regression test covering null timestamps, duplicate timestamps, and visibility/meta filtering behavior.
File summaries
| File | Description |
|---|---|
packages/core/src/query.ts |
Adds bounded, index-assisted temporal context retrieval with a scan fallback and prepared-statement reuse. |
tests/query.test.mjs |
Extends search tests to validate neighbor selection around null and duplicate timestamps; aligns test DB with the production timestamp index. |
Review details
- Files reviewed: 2/2 changed files
- Comments generated: 1
- Review effort level: Lite
💡 Add a code-review agent skill or configure MCP servers for context-aware, tailored reviews. Learn more in the docs.
| const ctx = (indexedTimestamp ? indexedContext : scanContext) | ||
| .all(contextParams) | ||
| .map(withVisibility) | ||
| .sort((a: DbRow, b: DbRow) => a.timestamp < b.timestamp ? -1 : 1); |
There was a problem hiding this comment.
Fixed in 5d89ce8. Equal timestamps now return 0, null timestamps have an explicit null-first order, and non-null ISO timestamps remain ascending. The regression test now asserts the exact duplicate/null order instead of sorting UUIDs before comparison. Verification: query suite 32/32; old-SQL vs indexed-context differential 120/120 real hits with 0 mismatches.
6f58273 to
f0eaa08
Compare
|
Addressed Copilot's comparator-contract finding in 5d89ce8. Equal timestamps now return 0, null ordering is explicit, and the regression pins exact null/duplicate-timestamp order. Current-head verification: Linux 659/659 tests, typecheck clean, lint 0 errors (11 pre-existing/generated warnings), Windows query suite 32/32, and 120/120 real-hit old-vs-new context comparisons with 0 UUID/order mismatches. As a fork author I do not have upstream RequestReviewsByLogin permission; could a maintainer please re-request Copilot review? |
What and why
search()currently runs one temporal-context query per FTS hit. That queryorders every other message in the hit session by a
JULIANDAY()expression, soSQLite cannot use
idx_messages_ts; a hit in a 102,648-message session costsabout 206-235 ms just to collect six neighbors.
This change bounds that work without changing the result contract:
idx_messages_tsto take at most six rowsfrom each side of the hit (plus at most six null-timestamp rows);
JULIANDAY()distance androwid tie ordering;
search()call instead of once per hit.No schema, dependency, provider, visibility, or public API changes.
Reproduction and ablation
Windows 11, Node 24.16.0, immutable 3.33 GB SQLite snapshot with 1,627,293
messages. Each value is the median of five warm
search(query, { limit: 20 })runs. "Disabled" is current parent
fb4a8ef; "enabled" is this commit.索引indexerrorpythonobeliskEqOpThe six ordered hit-ID digests were identical with the optimization disabled
and enabled. An old-vs-new temporal-context differential over 120 real hits had
0 context UUID/order mismatches. The new regression test separately pins null
timestamps, duplicate timestamps, meta filtering, and inactive/hidden rows.
Verification
npm test— 659 pass / 0 fail on Ubuntu WSL2 (Node 24.14.1)npm run typecheck— 0 errors (root + app)npm run lint— 0 errors, 11 pre-existing/generated warningstests/query.test.mjsis matched bytests/*.test.mjs)The current-head query suite also passes 32/32 on Windows. Linux verification
was rerun after rebasing onto
fb4a8ef.Deliberately out of scope
post-FTS result context lookup.
idx_messages_ts(session_id, timestamp)issufficient.
app/files changed.