Skip to content

fix(observers): provide truthful packet metrics and freshness - #169

Open
n30nex wants to merge 4 commits into
MeshCore-Beacon:devfrom
n30nex:codex/beacon-observer-metrics
Open

n30nex wants to merge 4 commits into
MeshCore-Beacon:devfrom
n30nex:codex/beacon-observer-metrics

Conversation

@n30nex

@n30nex n30nex commented Sep 27, 2026 •

Copy link
Copy Markdown
Contributor

Observer monitoring separates broker presence from packet arrival: status and neighbor reports no longer advance packet freshness. The compatible legacy counter is explicitly documented. Activity responses include effective UTC bounds, generation time, stored total, last complete hour and latest retained packet time; optional until aligns comparisons. Migration 040 accounts for unknown payloads without inventing discarded signal samples.

Closes #168. Follows server #167. Parent 70c5c389f9886cb8d8c9d9d34462c805cad6fc6b, head 39c9d185087dcf6bc5aede8f8d1e86c61c1365c9. Focused diff.

Validated after integration with accepted #177–#180: local build/format/vet/tests and published-head CI pass. The composed Pi build 2ed2e031 includes all current server candidates and independent #166; its full PostgreSQL suite passes, including route-lock batching/cancellation and queue behavior. An isolated 504-scope replay preserved 100 packets, 800 observations, 100 decrypted messages and 2,400 opt-in live hearings from 3,200 inputs while a later maintenance batch was blocked. This is fixture evidence, not a universal delivery guarantee.

The backend is deployed on the Pi preview with exact source and changelog, retaining configuration, schema042 and rollback. All earlier review features remain included. Owners retain merges and production release. AI-assisted implementation continues under the contributor’s standing project authorization.

@n30nex
n30nex requested a review from 446564 as a code owner September 27, 2026 02:11
@n30nex

n30nex commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author

@MrAlders0n, this observer-release slice is ready for your/Claude's review at 60ed339c. Server order is #167 → #169; #166 remains independent. Native PostgreSQL migration/unknown-type/duplicate/freshness tests and exact-head CI/CodeQL pass. The schema039 database, binary/config and private off-Pi backup are retained for rollback.

The combined Pi preview is live at https://canadaverse.org/beacon-dev/?tab=Observers with server 88c2c10c / web 42ae09b7, preserving #166/#167/#75. Desktop, 390-pixel phone, English/French, keyboard and navigation checks pass; the changelog and exact source are at https://canadaverse.org/beacon-dev/source.html. Build CI passes (web CodeQL remains skipped). No merges or production switch were performed.

Docs #5 carries the observer-first plan and subsequent connected-investigation work. The proposed public per-IATA MeshMapper scope catalogue is drafted separately, with manual scopes retained; its endpoint/schema is not yet published. The captured historical 1,810 counter difference remains unexplained, not a measured packet-loss claim.

@n30nex

n30nex commented Sep 27, 2026

Copy link
Copy Markdown
Contributor Author

@MrAlders0n, ready for your / Claude's review at 91b21995. The declared server stack has been refreshed for accepted #170; this PR body now links its current focused diff and dependency. Full native Pi/PostgreSQL tests pass in composed server 99e623c5, with every existing candidate included. Exact-head CI passes.

The preview/source now runs server 99e623c5 / web 3b3abdcc. Migration 041 was tested on a restored copy with unchanged row fingerprints; prior database/binary/web recovery is retained. Server merge order is #167 -> #169 -> #172; #166 remains independent. No upstream merge or production switch was performed.

@n30nex
n30nex force-pushed the codex/beacon-observer-metrics branch from 91b2199 to 39c9d18 Compare September 28, 2026 22:13

@MrAlders0n MrAlders0n left a comment

Copy link
Copy Markdown
Member

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

  1. Migration 040. Please fold it into 039 (see the #167 review). On prod both would run back to back, and the ingest pause doubles.
  2. The latest partial bucket is gone. until = now.Truncate(interval) together with heard_at < until drops the current partial bucket, so web's 15m, 1h and 6h charts lag by up to one bucket, and a quiet observer looks dead at the right edge. I'd rather keep the partial bucket when until isn't supplied, and truncate only when a client asks for a specific until.
  3. Cache key. internal/cache/reader.go:376 keys on the raw until, while the store truncates it. A client sending Date.now() never hits the cache and writes a new key every request. Truncate before building the key.
  4. summary (lastCompleteHour, latestRecordedAt) ignores until. Either document that or leave summary out when until is set.
  5. Nits: the ObserverActivity doc comment now sits above ObserverActivitySummary. Please put the new query in db/queries/queries.sql rather than a separate file.

The NaN/Inf filtering and broker freshness changes look good.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

Observer dashboard needs reliable packet metrics and separate traffic freshness

2 participants