feat(db): retain 30-day analytics independently of packet expiry

[n30nex]

Raw packet expiry used to erase longer analytics during materialized-view refresh. Archive compact hourly cohorts atomically with deletion and combine them with live rows for 30-day summaries. The initial migration now includes unknown observer payloads, restores both hourly Signal/Paths indexes, uses non-null/default-zero count columns and serializes competing cleanup runners before their shared upserts. Hourly IATA statistics move from seven days to 30 days with UTC buckets.

Closes #165. First PR in the ordered server stack. The unmerged observer correction is folded into 039; no back-to-back 040 rebuild or preview-specific repair INSERT is shipped. The existing Pi received a separately tested operator repair.

Parent db30c9b573357990c41166292e7cf42785c92cc4, head 0c19e95132760eda1bdcf550ae1b01865bbf3c41. Focused diff.

Published-head build/CodeQL checks and the native Pi Go/PostgreSQL suite pass. The combined server 397d76b3 is running on the development Pi. Its 3,200-input/504-scope replay passed with zero fixture drops; that does not establish universal production losslessness. Release scope and recovery / running source. The web release cut is being validated separately; Atlas is deferred until after web 1.3.2.

Maintainers retain merges, tags and production release.

[n30nex]

@MrAlders0n, please review this with Claude, especially migration 039, archive/delete atomicity and the preserved counting semantics. It is independent of #166 and can merge in either order. Both source candidates are being composed and tested on the Pi preview with 72-hour raw packets, 30-day summaries and the changelog. The archive cannot recover already-purged history. The contributor account cannot submit a formal reviewer request, so this comment is the review request.

[n30nex]

Deployment validation is complete. The Pi preview now runs the exact composed server a8394f10 (independent #166 and #167 on dev 91b4b45) with web 3a18e6d1 (#75). Changelog and corresponding source.

Raw packets are configured for 72 hours, hourly analytics for 30 days, telemetry for 720 hours. Both brokers connected; public Traffic/Signal/Paths and packet-candidate UI checks pass. Native full Go/PostgreSQL tests and all 874 web tests pass; exact-head CI is green. A restored data copy preserved all eight view counts after raw deletion in a rolled-back test. Original schema038 DB and the prior build remain available for rollback. Already-purged history is unavailable; a month of accumulated traffic is not yet present.

No PR depends on the other server PR; they can merge in either order. The shared roadmap is updated in docs #5.

[n30nex]

@MrAlders0n, ready for your / Claude's review at 02743704. The declared server stack has been refreshed for accepted #170; this PR body now links its current focused diff and dependency. Full native Pi/PostgreSQL tests pass in composed server 99e623c5, with every existing candidate included. Exact-head CI passes.

The preview/source now runs server 99e623c5 / web 3b3abdcc. Migration 041 was tested on a restored copy with unchanged row fingerprints; prior database/binary/web recovery is retained. Server merge order is #167 -> #169 -> #172; #166 remains independent. No upstream merge or production switch was performed.

[MrAlders0n]

Nice design. Locking with SKIP LOCKED, archiving and deleting in one transaction is the right shape. Two things before merge:

1. Dropped indexes. Migration 039 drops and recreates mv_signal_stats_hourly and mv_path_stats_hourly but only restores the unique indexes. idx_mv_signal_stats_hourly_hour (035) and idx_mv_path_stats_hourly_hour (036) are lost. Stats and paths queries without an IATA filter only on hour, so they would scan up to 30 days of rows. Please recreate both.
2. Observer view built twice. 039 builds mv_observer_activity_hourly, and #169's 040 drops it and aggregates the same 30 days again. Neither migration has run upstream, so please fold 040's COALESCE(payload_type,-1) view and archive-function change into 039. The one-off repair INSERT is only needed on your preview DB; fix that database by hand.

Smaller:

• Consider pg_advisory_xact_lock at the top of archive_delete_packets. Two cleanup runners upserting overlapping (iata, hour) keys can deadlock.
• Archive count columns could be NOT NULL DEFAULT 0 to avoid pointer types in models.go.
• mv_hourly_iata_stats moves from 7 to 30 days and to UTC buckets. Fine, but note it in the PR description.

[n30nex]

@MrAlders0n / Claude: addressed at 2ab36356.

Both dropped hourly indexes are restored. Unknown payload handling is folded into 039, and #169 removes 040 entirely, so production does not rebuild the observer view twice. Preview-only repair SQL stays outside the migration. I also added the transaction advisory lock and NOT NULL/default 0 counts, and documented the 30-day UTC IATA window. Native coverage checks index presence, competing cleanup transactions, NULL-payload preservation, failure/retry and archive stability after deletion.

Published-head CI and the complete combined native Pi/PostgreSQL suite pass. The Pi runs server a35cba1d / web e1133ab5, retaining every review candidate. Windows/Pi web build/lint and all 940 tests pass. The isolated 3,200-input/504-scope replay preserved expected data/events with zero fixture drops; this is not a universal production losslessness claim. Validation, exact heads and recovery / running source.

Please take another look when convenient; no upstream merges were performed.

[MrAlders0n]

Thanks, verified all of these. One small follow-up, fine after merge: the README says mv_hourly_iata_stats covers 30 days in UTC buckets, but not that it used to be 7 days (001). Please add a line saying so, since /stats/observations?since= older than 7 days now returns data.

[n30nex]

@MrAlders0n / Claude: September 30 follow-up at 0c19e951.

Added the explicit seven-day to 30-day change in the README, including the effect on older /stats/observations requests.

Published-head build/CodeQL checks and the native Pi Go/PostgreSQL suite pass. The combined server 397d76b3 is running on the development Pi. Its 3,200-input/504-scope replay passed with zero fixture drops; that does not establish universal production losslessness. Release scope and recovery / running source. The web release cut is being validated separately; Atlas is deferred until after web 1.3.2.

Please re-review the current head when convenient. No upstream merge or tag was performed.