perf(store): avoid eager per-transmission observation dedup maps #411

Context

Assessment baseline: dborup/CoreScope master 4a5ec5e231dd97ed7a4392e50e5f409f4dc5fab0 (13 October 2026). Recheck current master and competing work before implementation.

The fork eagerly keeps observation-key and observer-set maps per transmission, including allocated concatenated keys. Most transmissions have few observations, making those permanent maps a potential large RAM cost.

Source for later comparison

Verified from GitHub's PR metadata and complete changed-file listing on 13 October 2026. The identifiers below are literal text, not hyperlinks or cross-repository references.

Source repository: Kpa-clawbot/CoreScope
Pull request number: 2181
PR head commit SHA: 8d34e814d3a4e714d88b7c23b5bae407333a630d
Upstream merged SHA (merge_commit_sha): dc7ebcd6f546f594362eaf8a14e19258fb6169e8
Files changed in this PR:
  - cmd/server/chunked_load.go
  - cmd/server/eviction_test.go
  - cmd/server/obs_dedup.go
  - cmd/server/obs_dedup_test.go
  - cmd/server/store.go
  - cmd/server/tracked_bytes_test.go

These are historical upstream comparison sources, not proof of equivalence with our fork and not a request to cherry-pick blindly. Recheck our current implementation and preserve fork-specific behavior.

Proposed work

• Measure current memory and observation-count distributions before choosing a bounded small-list versus map threshold.
• Evaluate bounded scans for small observation lists and structured-key maps only when justified; do not change the observation identity contract.
• Apply the design consistently across hot load, background load, ingestion, upsert, eviction and rebuild paths.
• Preserve unique observer counts, same-row route updates, resolved relay indexes and trackedBytes/maxMemoryMB accounting.
• Add equivalence, duplicate/threshold/eviction and race tests; benchmark adversarial high-observation transmissions as well as normal data.

Acceptance criteria

A measured memory reduction on our fork without changing retained observations, unique counts, route evidence or memory-budget enforcement. CPU work stays bounded.

Existing work / dependencies

Keep the fixes tracked by #113 and #158 intact; this is not another resolved-relay dedup bug fix.

Scope

This issue records evaluation/follow-up work in our fork. It does not authorize deployment, production/staging changes, mail delivery, automatic key promotion or other external actions. Preserve fork-specific correctness, bounded work and the read-only measurement-server database contract; any separate account database exception requires explicit review.

Source performance numbers are not measurements of our deployment. Any claimed improvement requires our own benchmark and an independently reviewed implementation.