Skip to content

perf(search): simplify retrieval queries and lifecycle checks - #2095

Merged
hidb4ai merged 10 commits into
masterfrom
perf/search-lifecycle-queries
Oct 20, 2026
Merged

hidb4ai merged 10 commits into
masterfrom
perf/search-lifecycle-queries

Conversation

@PsiACE

@PsiACE PsiACE commented Oct 20, 2026 •

Copy link
Copy Markdown
Member

Which issue or RFC does this PR close?

Related: RFC 1417 — Topic Memory.

Rationale for this change

Reduce retrieval scans and duplicate reads while preserving Scope, exact revision, authority, and lineage semantics.

What changes are included in this PR?

  • Use sqlite-vec KNN and native ANN for bounded Atomic search; keep complete enumeration exact.
  • Rank Topic detail by parent; retain title HNSW and existing detail graphs.
  • Qualify FTS candidates before truncation, reuse selected bodies, and batch authority reads and processing work.
  • Give indexes explicit lifecycle ownership; reuse versioned Experience/Skill text on restart and preserve Skill reactivation.
  • Update behavior tests and workflow documentation; fix the installed Kimi Scope helper path. RFCs are unchanged.
             Before                         After
Search       repeated audits/counts          eligible candidates
             -> repeated authority reads    -> batched authority checks

Topic        raw chunk pool -> collapse     parent-level ranking -> Top-K

FTS          LIMIT -> coverage              qualified prefix -> exact fallback

Processing   per-item reads/embedding       batch reads + scoped embedding reuse

FTS startup  rescan bodies/packages         reuse versioned text + native FTS

Are there any user-facing changes?

No HTTP schema or migration-file changes. SQLite Atomic uses a derived vec0 helper; incompatible Atomic vectors require repair before startup. Experience/Skill text refreshes on version changes or explicit repair. Internal adapters now declare index capabilities and return ranked rows; public Topic callbacks must return complete exact Artifacts.

How was this change tested?

  • make check and make harness-check passed.
  • Interface/result tests: 93 passed. Startup/lifecycle tests: 22 passed; real OceanBase restart/upgrade: 2 passed.
  • SQLite increment vs 6c9fec89: 500 Artifacts per Family, 20 samples/case. Experience/Skill search SQL: 5→4 at K=10/100, 7→5 at K=200; complete responses unchanged.
  • SQLite FTS-only warm initialization: 135.59→0.91 ms, five-sample medians at 500 Experience + 500 Skill. This excludes full server startup.

The chart shows the master-baseline retrieval matrix; it is not a current-head full-matrix rerun. The checks above cover the additional interface and startup changes.

Retrieval performance against master

AI usage statement

Codex assisted implementation and testing; GPT-6 Astra (xhigh) assisted design review.

Let indexes own startup validation and declare supported search families. Share Experience/Skill candidate reads and materialize exact results without rereading bodies; require complete Topic results in the retrieval snapshot.
Record completed refreshes in the existing lexical version table. Reuse stored text and native indexes on warm startup, and retain explicit projection repair. Keep inactive Skill heads indexed so reactivation remains searchable after a restart.
@PsiACE
PsiACE marked this pull request as ready for review October 20, 2026 04:10
Copilot AI balanced review requested due to automatic review settings October 20, 2026 04:10

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@hidb4ai

hidb4ai commented Oct 20, 2026

Copy link
Copy Markdown
Contributor

Please resolve the merge conflict with the latest master in e2e/bub/tests/test_harbor_job_config.py. The conflicting changes overlap with #2090, which has already been merged. After resolving the conflict, please rerun the affected Kimi Code harness tests and confirm CI passes before merging.

Keep the upstream Kimi installation test normalization; both branches already contain the same installed Scope helper fix.
Copilot AI balanced review requested due to automatic review settings October 20, 2026 05:08

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@AlexStocks AlexStocks added the 1.4.1 v1.4.1 release tracking label Oct 20, 2026
Comment thread src/powercontext/builtin/persistence/sqlite/topic_memory_index.py
Require SQLite 3.35.0 for materialized retrieval CTEs before creating the engine or accepting writes. Report the linked library version, preserve existing storage on rejection, and document the backend minimum.
Copilot AI balanced review requested due to automatic review settings October 20, 2026 11:38

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

Copilot AI balanced review requested due to automatic review settings October 20, 2026 12:41

Copilot AI left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot was unable to review this pull request because the user who requested the review has reached their quota limit.

An organization owner can choose the repository organization as the billing source for reviews requested by members with a paid Copilot license. Learn about Copilot code review billing.

@hidb4ai hidb4ai left a comment

Copy link
Copy Markdown
Contributor

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

LGTM

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

1.4.1 v1.4.1 release tracking

Projects

None yet

Development

Successfully merging this pull request may close these issues.

5 participants