Skip to content

Integrate persistent SQLite indexes with native server reads #141

Description

@DandyLyons

Goal

Replace repeated collection discovery with a shared native indexing service used directly by the CLI and server. Part of Phase 2.5 in #93; depends on #138, #139, and #140.

Requirements

  • Integrate indexed filesystem reads behind existing repository/RecordStore boundaries; keep files authoritative.
  • Publish consistent refreshed data snapshots without unexpectedly changing the immutable EndpointPlan.
  • Define explicit behavior for resource configuration changes and failed refreshes.
  • Add pagination, filtering, and limits with deterministic behavior; expose full-text search only when enabled under Add optional body indexing with single-copy FTS5 storage #146. Declare search capability in configuration and the generated OpenAPI contract, with explicit behavior when unavailable.
  • Generate new read parameters/responses from the same plan and test OpenAPI parity.
  • Preserve identity collisions, overlapping membership, and visibility of invalid rule-selected records.
  • Indexing a rule/type never automatically exposes an HTTP resource.
  • Use direct library calls rather than spawning the CLI.
  • Consume Bound collection refresh memory and reuse extraction across scopes #147's bounded refresh/publication service and Use runtime-selected JSONB with a JSON-text compatibility mode #148's encoding policy. Do not introduce ripgrep subprocesses into server reads.
  • Preserve Build the immutable server read snapshot and generic record representation #89's body-bearing record representation in metadata-only mode through bounded authoritative-file reads, checked against indexed revisions. Specify refresh or explicit failure when files change; never silently combine old indexed metadata with a newly read different body version.
  • Bound page materialization and response buffering. Do not rebuild a corpus-sized in-memory body cache in immutable read snapshots. Preserve the existing RecordStore and EndpointPlan boundaries.

Acceptance criteria

  • CLI and server reuse indexing semantics and persistence.
  • Concurrent reads see consistent generations as updates occur.
  • Existing server contract fixtures retain their semantics.
  • Pagination/filter/search and generated OpenAPI agree.
  • Refresh failures and configuration changes are explicit and tested.
  • Metadata-only and FTS-enabled resources expose their actual search capabilities consistently with OpenAPI.
  • Metadata-only body reads preserve record/revision consistency under concurrent file changes.
  • Large collections and body-bearing pages use bounded materialization rather than caching the entire corpus in memory.

Boundaries

This is an indexed filesystem adapter, not a canonical SQLiteRecordStore. Portable Core and the later Workers runtime do not require native SQLite extensions. UI editing is outside #93.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions