perf: persist map-nav scoring index for large corpora

[suguanYang]

Summary

• Batch lazy MAP-NAV snapshot payload reads to bounded keyset queries.
• Persist revision-pinned MAP-NAV lexical scoring units and token frequencies.
• Use the persisted scorer when the index is complete; fall back to the existing legacy scorer for incomplete or inconsistent indexes.
• Preserve exact eager/legacy scoring and retrieval result semantics.
• Document the required existing-revision backfill operation for deployment.

Verification

• Focused retrieval/map-index contract tests: passed.
• 60k large-corpus contract: passed.
• Ruff: passed.
• Pyright: 0 errors.
• Local 600k HTTP differential test: indexed median 35.671s vs forced legacy median 85.771s; exact response payload equality, including scores, ordering, references, and evidence.

DevOps Operation

The Alembic migration creates the derived index tables. Existing document revisions must then be backfilled before they receive the faster scoring path. Until backfill completes, retrieval remains quality-preserving and uses the legacy fallback.

Run this as a one-off task/container using the newly deployed API image and the production database secret. Do not run it inside the long-lived API task.

# 1. Inventory current revisions (read-only)
python /app/scripts/backfill_map_unit_indexes.py

# 2. Optional canary: backfill one document, then verify retrieval
python /app/scripts/backfill_map_unit_indexes.py \
  --document-id <document-id> \
  --apply

# 3. Backfill all current revisions
python /app/scripts/backfill_map_unit_indexes.py --apply

The script commits each document revision independently and is safe to rerun. New or republished documents build their index automatically during publication.