feat(analytics): classify adverts from durable route evidence

[n30nex]

Fixes #2041; addresses the mixed-route review and supplies #2085's authoritative node classification.

An advert hash excludes routing. Observation UPSERTs can erase an intermediate route, so the first frame and surviving rows are insufficient. The ingestor now persists at most two route-evidence rows per retained advert. Repeated evidence does not append rows or advance its sequence; retention removes evidence with its transmission.

A bounded independent feed updates the read-only server even when observation IDs stay unchanged. Cold/chunk/live loads agree on flood, zero-hop, mixed or unknown. The chart keeps existing count/score formulas, including zero-relay rows. Node details expose the same advert_kind.

Backfill runs after ingestion readiness in cancellable, resumable 500-row batches. Live writes preserve unbackfilled observations before overwrite. It recovers surviving evidence only; the chart states the historical limitation. Time windows filter counts/scores, while classification uses known hash history.

Validation

Red 6f8066d: assertion failures. Tests cover both ingestion orders, overwritten middle frames, restart, retention, backfill rollback/resume, loading, cache updates and unchanged totals. Real ingestor/API/desktop/mobile proof, all 183 frontend suites, lint and focused race checks passed.

O(n) cached analytics: 30K-packet seven-day median 17.69 → 21.96 ms versus 6f8066d, unchanged ~1.75 MB/165 allocations. Bulk masks run outside the store lock; invalidation is per batch. No speedup claim.

Correction red 6a87e0a; green 8523576. Full CI passed at 8523576 (run), including Linux Go, race, browsers, coverage, both container architectures and ARM smoke tests. Local Windows full-ingestor testing required symlink privilege. No configuration changes.

[dborup]

The test-first approach looks useful, but the implementation should avoid inheriting the first-ingested-route limitation identified in #2085.

CoreScope deduplicates transmissions by packet hash, while later observations of the same packet may contain different route evidence. The stored transmissions.route_type / raw_hex therefore represents only one ingested route and is not sufficient to decide that an advert was exclusively flood or zero-hop.

Please consider accumulating route evidence across observations and representing a mixed case when both flood and zero-hop routes have been observed. Otherwise the Relay Airtime Share result can depend on observation arrival order rather than the complete route evidence.

It would also be valuable to include a regression test where the same packet hash is observed first through one route class and later through the other, in both ingestion orders.

[n30nex]

Review feedback addressed (commit 8523576, following 9ff1d29):

1. Persisted bounded route evidence survives same-observation overwrites in both ingestion orders, restart and retention.
2. A separate bounded feed keeps node classifications and Relay Airtime Share consistent, including mixed routes and unchanged totals.
3. Legacy observations are preserved before live overwrites during backfill; failures leave the old frame intact.
4. Added history-limit explanations and upgrade-race regressions.

All 183 frontend suites, full server tests, parent race checks and real-ingestor desktop/mobile validation passed. Three independent reviews are clear. Exact-head Linux CI remains required; the local full ingestor suite needs Windows symlink privilege.

[efiten]

Reviewed against upstream/master at 9eb30988. This is careful work: the read/write invariant holds, storage is bounded at two rows per retained advert with FK cascade eviction, the interface{} count does not increase, and the tests are behavioural rather than implementation-shaped. advert_route_evidence_test.go:12-118 asserting the observation really was overwritten before asserting the evidence survived is exactly the right shape.

I verified the protocol decoding in internal/packetpath/advert.go line by line against firmware/docs/packet_format.md and src/Mesh.cpp, including the hashSize == 4 rejection and the sendZeroHop marker. It is correct.

Two things should be settled before this merges.

1. It adds a second advert classifier next to the one it was written to fix, in the same response

cmd/server/advert_stats.go:63-67 still classifies from transmissions.route_type alone:

WHERE from_pubkey = ? AND payload_type = ? AND route_type = ? AND first_seen >= ?

That is the first-ingested-route bias. It feeds flood_advert_count_7d, and routes.go:1664 writes that into the same payload as the new field:

writeJSON(w, NodeDetailResponse{
    Node:          node,           // carries flood_advert_count_7d
    RecentAdverts: recentAdverts,  // carries advert_kind
})

For any mixed advert those two contradict each other: route_type=2 excludes it from flood_advert_count_7d while advert_kind reports mixed. AGENTS.md is explicit that one rule gets one implementation.

Either migrate CountFloodAdvertsForNode onto the evidence table here, or drop advert_kind from the node API and say in the body that #2085's consumer is deliberately left for a follow-up. As it stands advert_kind on that endpoint has no consumer either: git grep advert_kind -- public/ hits only analytics.js:547-550, and public/nodes.js is untouched.

2. A failed write to an analytics side table now drops the advert packet

cmd/ingestor/db.go:1129-1132:

if _, err := s.stmtInsertAdvertEvidence.Exec(txID, bit, txID, bit); err != nil {
    s.Stats.WriteErrors.Add(1)
    return isNew, fmt.Errorf("record advert route evidence: %w", err)
}

Sixty lines later the observation insert itself is explicitly non-fatal (db.go:1191: log.Printf("[db] observation insert (non-fatal): %v", err)). So the side table is treated as more critical than the observation it describes. On SQLITE_BUSY or a disk-full mid-run, that advert loses its observation, its resolved_path, the touchRelayNodesLocked call and the last_seen bump, where before it would still have landed.

TestAdvertRouteEvidenceWriteFailurePropagates asserts this deliberately, so it is a design decision rather than an oversight, which is why I am asking rather than patching. The counter is already incremented; what I would change is returning instead of carrying on.

Four smaller things

The perf number measures the case least affected. relay_airtime_share.go:157-162 adds an unconditional pre-pass over all of s.packets sized len(s.packets). Previously seenHash filled lazily inside the window-filtered loop, so the reporting window bounded it; now it does not. The quoted 17.69 to 21.96 ms is a seven-day window, where the pre-pass is nearly free because the window is the whole store. The analytics page defaults to a short window, which is the expensive case and is not measured. AGENTS.md rule 0 asks for the worst case.

Related, and I have not measured it: pollAdvertEvidence streams 500 rows per tick and calls invalidateAdvertEvidence whenever anything changed. Starting the server while the ingestor backfill is still running means dropping every relay-airtime-share|* entry on each tick for the duration, each subsequent miss paying the full pre-pass. What is the observed catch-up time on a production-sized database?

The durability claim needs scoping. "Persisted bounded route evidence survives same-observation overwrites in both ingestion orders, restart and retention" is true for traffic ingested after the upgrade, and I confirmed that: the bit is derived from the incoming frame and written on every InsertTransmission regardless of isNew. For pre-upgrade history advert_route_evidence.go:52-56 recovers only frames that still exist, so any UPSERT that overwrote a frame within one observation key is gone. An advert heard flood, then zero-hop, then flood by a single observer before the upgrade classifies as flood. You do disclose this at internal/packetpath/advert.go:47 and in the UI note; the sentence in the PR comment does not carry the qualifier.

Backfill wall clock. The backfill scans both transmissions and observations in 500-row batches with a 10 ms yield, taking writerMu once per batch. On an 11M-row observations table that is roughly 22,000 writer acquisitions and at least 220 s of sleep alone. It is bounded, cancellable and resumable, but until it finishes every advert reads Other adverts in the chart, which looks like a fault. What did it measure at on a production-sized DB, and is it worth saying in the caveat note that the number improves on its own?

Unrelated change bundled in. cmd/server/db.go:1296 changes if limit <= 0 to if limit <= 0 || limit > 20. Nothing about route evidence needs it, and it silently clamps a caller's request rather than honouring or rejecting it. Only production caller passes 20, so there is no live impact, but rule 6 asks for one logical change per commit.

One question

Can a direct-routed ADVERT reach the air with path_len == 0 after removeSelfFromPath (firmware/src/Mesh.cpp:334-342) consumed its last hop? On the wire that is indistinguishable from a zero-hop send. I could not find a firmware path that direct-routes an advert carrying a path, so this is probably unreachable, but I could not prove it. If it is unreachable the reasoning is worth a line in the comment; if it is not, AdvertZeroHop is over-claimed.

Minor, no action needed

• advertEvidenceForIDs does not validate SUM(bit) the way pollAdvertEvidence does at advert_route_evidence.go:145-148. A corrupt bit=3 row gives the load path mask = 4, and AdvertKind(4) returns "other" silently, while the poll path errors.
• The two new Playwright tests mock /api/analytics/relay-airtime-share entirely, so they prove nothing about the server producing advert_kind. The Go tests cover it; noting the E2E does not close the loop.
• tests/unit/test-frontend-helpers.js:2076 extracts the renderer by string substitution on registerPage('analytics',. It fails loudly if the literal moves, so it cannot pass on broken code, but it is brittle.