Feature request: Reference-to-many observer comparison with complete-window packet overlap #122

Summary

Extend the existing Compare page with a reference-to-many observer comparison: select one reference observer and compare the traffic seen by other observers over the same time window.

This should be a descriptive troubleshooting tool, not an automatic RF-health score. Keep SNR comparison and health classification out of the first version.

Related upstream proposal: Kpa-clawbot#671

Existing functionality and motivation

The current Compare page already compares two observers and shows asymmetric overlap percentages. However, it requests at most 10,000 packets per observer for the last 24 hours and computes the intersection in the browser. Busy observers can therefore be compared using incomplete subsets.

Reference implementation inspected: public/compare.js.

Build on this page rather than introducing a separate top-level analytics page. Preserve the existing two-observer workflow and its deep links.

Proposed MVP

1. Reference selection and comparison table

• Select one observer as the reference.
• Offer 1-hour, 6-hour, and 24-hour windows initially.
• Compare the reference with other eligible observers in a single table; respect existing observer visibility and filtering policies.
• Show each observer name, unique received transmission count, shared count, reference-only count, other-only count, and overlap percentages in both directions.
• Allow drilling into a particular pair to inspect shared and exclusive packets. Detail lists may be paginated, but must use the same window and filters as the summary.
• Reflect the mode, reference, time window, and relevant filters in the URL so results can be shared/bookmarked. Remember the selected reference locally.

This applies to observers that supply reception data to CoreScope, not automatically to every repeater appearing on the map.

2. Exact and explicit overlap semantics

Let R be the set of unique stored transmissions received by the reference, O the corresponding set for another observer, and S = R intersection O.

Display:

• Shared count: |S|
• Reference-only count: |R| - |S|
• Other-only count: |O| - |S|
• Percentage of reference traffic also seen by the other observer: 100 * |S| / |R|
• Percentage of the other observer traffic also seen by the reference: 100 * |S| / |O|

For example, reference = 1,000, other = 800, shared = 600 means 60% of reference traffic and 75% of other traffic. Neither percentage is a health score.

Requirements:

• Compute aggregates server-side from all retained eligible observations in the requested window, without inheriting packet-list pagination limits or an in-memory retention cutoff.
• Use one explicit UTC interval [start, end) for both sides, based on observation timestamps, not only transmission first_seen. Return the effective interval in the response.
• Count each transmission_id once per observer, even when multiple paths/observations exist. These are unique stored transmissions, not a count of every physical radio emission.
• Use explicit RX observations; exclude TX and do not silently treat an unknown direction as RX.
• Return an unavailable percentage when its denominator is zero rather than presenting a misleading 0% score.
• Do not silently fall back to an older window when the reference has no data.

A bulk endpoint such as /api/analytics/observer-comparison?ref=...&hours=24 is a possible implementation; finalize its response contract before building the UI.

3. Honest interpretation and data-quality messaging

Show counts and known coverage limitations alongside percentages. Distinguish no data from zero overlap in a valid dataset. Retention limits, reporting gaps, or unknown radio compatibility must not be presented as proof of poor reception or full-period uptime.

Low overlap may reflect different locations, frequencies/SF settings, routes, downtime, or MQTT reporting gaps. A reference observer is a comparison baseline, not ground truth.

Do not introduce the upstream proposal's fixed red/yellow/green 30%/60% thresholds or labels such as healthy/degraded/failing in this MVP.

Explicitly deferred

• SNR/RSSI delta analysis. Shared packet content does not guarantee the same last-hop transmitter or radio conditions; naive joins can also multiply observations and bias the result.
• Automatic antenna-failure or RF-deafness diagnoses, health scoring, and alerts.
• Multi-reference consensus, map overlays, and 7-day comparisons.
• Historical radio-configuration tracking or a new telemetry schema.

A useful later extension is overlap over time against an observer's own baseline, with coverage/radio-change caveats, rather than universal health thresholds.

Performance and implementation constraints

• Keep analytics reads in the read-only server; any necessary index/schema changes belong in the ingestor.
• Use bounded, cached/coalesced bulk aggregation rather than one API call per observer or downloading every packet to the browser.
• Bound supported windows, request work, and cache cardinality. Never silently truncate the aggregate population to meet a limit; return an explicit limitation/error instead.
• Inspect query plans and benchmark realistic multi-million-observation datasets before enabling this on a live instance. Upstream toy-dataset estimates are not sufficient evidence.
• Document any new operational limits/settings and track customizer exposure where appropriate.

Acceptance criteria and validation

• Existing pairwise comparison remains available; the new reference-to-many mode has shareable state.
• Summary counts include eligible records beyond the existing 10,000-packet cap and do not change with detail pagination.
• Both directions of overlap are clearly labelled with their denominators/counts.
• Tests cover empty sets, zero/perfect/partial overlap, asymmetric counts, RX/TX/unknown direction, repeated observations on different paths, and observations whose transmission was first seen outside the window.
• Tests cover exact start/end boundaries, visibility filtering, unavailable reference data, and incomplete coverage messaging.
• Browser tests cover selection, window changes, stale responses, sorting, drill-down, reload/deep links, empty states, and narrow screens.
• Query-plan/benchmark evidence demonstrates bounded impact at realistic scale.
• No SNR comparison, automatic health classification, alerting, packet-retention changes, or data deletion is introduced.