Summary
Add incremental indexing at save time so new entries update term/frequency state without full corpus recomputation.
Motivation
Save-time processing must stay fast (<200ms typical) while keeping analysis state query-ready. Incremental updates avoid requiring full cf analyze runs between saves and queries.
Scope
In scope:
- Hook incremental analysis into
save_snapshot() for cf save and cf pre-compact
- Tokenize only the new entry
- Update SF/frequency state for affected terms only
- Keep incremental state consistent with batch recomputation semantics
- Likely files:
crates/core/src/engine.rs, crates/analysis/src/* incremental methods
Out of scope:
- Full corpus recompute path (
cf analyze)
- Save-time extraction/classification beyond agreed incremental subset
Acceptance Criteria
Dependencies
Complexity
Medium — correctness and performance constraints must both hold under incremental updates.
Summary
Add incremental indexing at save time so new entries update term/frequency state without full corpus recomputation.
Motivation
Save-time processing must stay fast (<200ms typical) while keeping analysis state query-ready. Incremental updates avoid requiring full
cf analyzeruns between saves and queries.Scope
In scope:
save_snapshot()forcf saveandcf pre-compactcrates/core/src/engine.rs,crates/analysis/src/*incremental methodsOut of scope:
cf analyze)Acceptance Criteria
Dependencies
Complexity
Medium — correctness and performance constraints must both hold under incremental updates.