Bound collection refresh memory and reuse extraction across scopes

[DandyLyons]

Goal

Scale incremental indexing to hundreds of thousands of files and evaluate million-document workloads without retaining the entire changed corpus in memory. Part of Phase 2.5 in #93; follows #138 and coordinates with #139, #140, and #141.

Requirements

• Replace accumulation of all changed evaluations/bodies with bounded batches and disk-backed staging. Define limits in bytes as well as records and document handling of individual oversized files.
• Read and extract each unchanged file version once per refresh across overlapping scopes, while retaining independent type/rule assessments and context-dependent evaluation.
• Refresh persisted document content once per file version; avoid duplicate writes/indexing across scopes.
• Preserve generation checks, concurrent-reader consistency, cancellation, failed-scan retention, parse-error visibility, and publication semantics. Partial staged work must not be presented as a successfully refreshed generation.
• Define staging cleanup and recovery after cancellation or crashes. Keep staging out of collection discovery and watch feedback loops.
• Avoid moving the corpus-sized body cache into the server snapshot or another in-memory layer.
• Benchmark initial indexing, rebuild, unchanged refresh, small updates, and overlapping scopes in metadata-only and FTS modes. Record peak memory, elapsed time, database/staging size, and relevant hardware/corpus characteristics.
• Set explicit measured resource budgets during implementation. Cover 100,000 and one million documents with realistic size distributions; do not infer scalability from document count alone or a 1,000-document test.

Acceptance criteria

• Changed body payload is bounded by the batch/individual-file policy rather than total corpus size.
• Overlapping scopes reuse extraction without changing assessment semantics.
• Failure, cancellation, stale generations, and recovery do not publish incomplete work as current or falsely delete inaccessible files.
• Concurrent reads observe documented consistent generations.
• Staging files are cleaned up or safely recovered and excluded from discovery/watch events.
• Reproducible 100,000- and one-million-document benchmark results and resource budgets are documented.
• Focused regression tests cover bounded ingestion and overlapping-scope behavior.

Boundaries

Files remain authoritative and filesystem scans are not atomic snapshots. Query-result streaming belongs to #139; bounded server materialization belongs to #141. Preserve pending edits separately from disposable staged/cache data (#142).