perf(store): evaluate on-demand subpath counts instead of permanent indexes #413

Context

Assessment baseline: dborup/CoreScope master 4a5ec5e231dd97ed7a4392e50e5f409f4dc5fab0 (13 October 2026). Recheck current master and competing work before implementation.

The fork retains both spIndex and spTxIndex. An on-demand bounded counting design could reduce permanent RAM and maintenance work, but must preserve our Route Patterns contract and deterministic ranking. External performance figures are motivation, not measurements of our deployment.

Source for later comparison

Verified from GitHub's PR metadata and complete changed-file listing on 13 October 2026. The identifiers below are literal text, not hyperlinks or cross-repository references.

Source repository: Kpa-clawbot/CoreScope
Pull request number: 2163
PR head commit SHA: 8af9582fd11e6efe79b18642b54530a301c5f89d
Upstream merged SHA (merge_commit_sha): be6aeffe4583ba0b72c04c7d17d5bd2a6cb9e468
Files changed in this PR:
  - cmd/server/channel_analytics_test.go
  - cmd/server/coverage_test.go
  - cmd/server/eviction_test.go
  - cmd/server/index_ready_1008.go
  - cmd/server/neighbor_graph_test.go
  - cmd/server/pathhop_eviction_1908_test.go
  - cmd/server/relay_airtime_share_test.go
  - cmd/server/repeater_snapshot_1908_test.go
  - cmd/server/resolved_index_test.go
  - cmd/server/store.go
  - cmd/server/subpath_index_memory_test.go
  - cmd/server/subpaths_window_test.go

Source repository: Kpa-clawbot/CoreScope
Pull request number: 2182
PR head commit SHA: 1bb07135ea93218a2e63f63303ba1baecafd660d
Upstream merged SHA (merge_commit_sha): 8d39a9d4652d321c6c2fd52f8dd74c9ce0fda2bd
Files changed in this PR:
  - cmd/server/channel_analytics_test.go
  - cmd/server/chunked_load.go
  - cmd/server/coverage_test.go
  - cmd/server/eviction_test.go
  - cmd/server/index_ready_1008.go
  - cmd/server/neighbor_graph_test.go
  - cmd/server/pathhop_eviction_1908_test.go
  - cmd/server/relay_airtime_share_test.go
  - cmd/server/repeater_snapshot_1908_test.go
  - cmd/server/resolved_index_test.go
  - cmd/server/store.go
  - cmd/server/subpath_index_memory_test.go
  - cmd/server/subpaths_on_demand_test.go
  - cmd/server/tracked_bytes_test.go

These are historical upstream comparison sources, not proof of equivalence with our fork and not a request to cherry-pick blindly. Recheck our current implementation and preserve fork-specific behavior.

Proposed work

• Compare a smaller incremental design (remove the duplicate count map) with replacing both permanent indexes by request-time counting.
• Preserve time-window membership, unique-transmission semantics, detail ordering and privacy filtering.
• Preserve lexical tie-breaking BEFORE the top-N cut; a count-only heap must not select different tied paths through random map iteration.
• Assess collision-safe keys rather than accepting hash-only aggregation without an explicit correctness decision.
• Keep response caches and scans bounded and account memory accurately. Coordinate with security(api): bound public analytics response caches #370.
• Measure heap/RSS, request latency and ingest/eviction costs on our data, including cold/warm detail requests; add equivalence and deterministic-tie regression tests.

Acceptance criteria

Choose a design only after fork-specific measurements show a worthwhile trade-off. Repeated tied queries return the same paths, counts and ordering; no route/query semantics silently change.

Scope

This issue records evaluation/follow-up work in our fork. It does not authorize deployment, production/staging changes, mail delivery, automatic key promotion or other external actions. Preserve fork-specific correctness, bounded work and the read-only measurement-server database contract; any separate account database exception requires explicit review.

Source performance numbers are not measurements of our deployment. Any claimed improvement requires our own benchmark and an independently reviewed implementation.