Skip to content

feat: add incremental indexing during save operations #54

Description

@asvarnon

Summary

Add incremental indexing at save time so new entries update term/frequency state without full corpus recomputation.

Motivation

Save-time processing must stay fast (<200ms typical) while keeping analysis state query-ready. Incremental updates avoid requiring full cf analyze runs between saves and queries.

Scope

In scope:

  • Hook incremental analysis into save_snapshot() for cf save and cf pre-compact
  • Tokenize only the new entry
  • Update SF/frequency state for affected terms only
  • Keep incremental state consistent with batch recomputation semantics
  • Likely files: crates/core/src/engine.rs, crates/analysis/src/* incremental methods

Out of scope:

  • Full corpus recompute path (cf analyze)
  • Save-time extraction/classification beyond agreed incremental subset

Acceptance Criteria

  • Each save triggers incremental tokenization for the new entry only
  • SF counts update correctly for terms in the new entry
  • Typical save-path latency impact remains under 200ms
  • Benchmark test compares incremental update vs full recompute
  • Consistency test shows incremental state matches full recompute results

Dependencies

Complexity

Medium — correctness and performance constraints must both hold under incremental updates.

Metadata

Metadata

Assignees

No one assigned

    Projects

    No projects

    Milestone

    No milestone

    Relationships

    None yet

    Development

    No branches or pull requests

    Issue actions