Skip to content

Add optional body indexing with single-copy FTS5 storage #146

Description

@DandyLyons

Goal

Make body storage and full-text indexing opt-in while retaining efficient, single-copy FTS5 search. Part of Phase 2.5 in #93; follows #137 and #138 and coordinates with #139 and #141.

Approved design

  • Persist one indexing mode per index initially, avoiding conflicting policies across overlapping scopes.
  • New indexes default to metadata-only: persist metadata, provenance, memberships and diagnostics, but neither bodies nor an FTS index. Do not require an FTS5 capability probe in this mode.
  • Existing indexes retain their search capability during migration unless explicitly changed.
  • FTS mode retains exactly one cached body per document. Use an external-content FTS5 table referencing that copy, with correct transactional insert/delete/update triggers and stable row identity. A separate body table is not required; preserve the one-to-one metadata/body relationship.
  • Retokenize only when the body changes, not when only metadata or assessments change.
  • Define explicit configuration/CLI controls for enabling and disabling search, migration, rebuild, and removal of obsolete body/FTS storage. Document when disk space is reclaimed.
  • Search requests in metadata-only mode fail clearly; do not silently enable indexing or substitute different search semantics.
  • Files remain authoritative. Metadata-only mode is intended to compose with tools such as ripgrep; FTS mode provides indexed word/phrase/proximity search and ranking. Do not implement a competing regex engine.

Acceptance criteria

  • New metadata-only indexes contain no persisted body payload or FTS tables and work without FTS5 support.
  • Existing indexes migrate without unexpectedly disabling search.
  • FTS mode stores one body copy and passes insert/update/delete/rebuild and FTS consistency checks.
  • Metadata-only changes do not retokenize unchanged bodies.
  • Mode switches, interruption, and failed migration have explicit tested behavior.
  • Shared documents across scopes retain one body and correct independent membership.
  • CLI/configuration/help and external SQLite query documentation describe both modes and unavailable-search errors.
  • Pending edits are preserved through mode changes; coordinate with Add revision-checked index apply for pending file edits #142.

Boundaries

No mandatory ripgrep dependency or server subprocess integration. Core remains SQLite-independent. Streaming query output belongs to #139; server capability/OpenAPI behavior belongs to #141.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions