feat(ingestor): resolve the last hop from the observer, relay-only prefix index, backfill NULL resolved_path #188

Summary

Make the ingestor's path resolution use the observer as the anchor for the last hop, then walk backwards. Restrict its prefix index to relay roles, and backfill existing NULL resolved_path rows within retention.

Today about half of all relayed traffic gives no relay credit. Every non-advert observation with 1-byte hop hashes gets resolved_path = NULL, because the ingestor has no anchor for those packets. Since #182, the server indexes relay credit (traffic share, handleNodePaths, paths-through) only from the persisted resolved_path, so the gap is visible network-wide.

This is a deliberate scoring change. Traffic share and usefulness change for most repeaters, and the sum of traffic_share_score rises from about 15 to about 22 (estimate). Decided by the developer on 2026-10-03.

Relates to #184, #182, #158.

Background (from the #184 investigation)

• Measurements: 3,000 transmissions, 61,787 observations with a path, 10 windows over 44 h on staging. An offline replay of the ingestor's resolver matched staging's NULL/non-NULL status for 99.6 % of rows.
  • All non-advert observations with 1-byte paths were NULL: 21,021 of 21,021.
  • 2- and 3-byte paths are almost never NULL: 9 of 39,504.
  • A 1-byte prefix is shared by 7.3 nodes on average. Only 1 of the 256 buckets is unique, or 4 when counting relay roles only.
• Cause: resolvePathWithContext (cmd/ingestor/path_resolver.go ~188) anchors hop 0 on fromPubkey, which is only set for ADVERTs (db.go ~2640). Each next hop is anchored on the previously resolved hop. Without an anchor, an ambiguous prefix stays nil (resolveHopWithContext ~143). If every hop is nil, the row is stored as NULL. The observer is never used as an anchor.
• Firmware: a hop is the first 1–3 bytes of the forwarding node's pubkey, and each forwarder appends its own hash before retransmitting (firmware/src/Mesh.cpp ~349, Identity.h ~19–31, Packet.h ~79–83). The last hop in the path is therefore the node the observer heard the packet from, which is a direct neighbor of the observer.
• Companions in the index: the ingestor's prefix index contains all nodes, including companions (neighbor_builder.go ~342–365). 64 companion pubkeys are stored as relays on staging. The server's map only has repeaters and rooms (cmd/server/store.go ~7131).
• Start-up window: the ingestor drains its buffered ingest (cmd/ingestor/main.go ~318) before the prefix index and graph are primed (~487). Rows ingested in that window are resolved with a nil index and stored as NULL.

Proposed change

1. Observer anchor. When the forward chain from fromPubkey gives no unique candidate, resolve the last hop as the candidate that is a neighbor of the observer, if exactly one is. Then walk backwards: hop i is the candidate that is a neighbor of the resolved hop i+1, if exactly one is.
   • Keep the conservative rule: no unique candidate means nil.
   • No geo, GPS or observation-count tie-breaks in persisted data (fix(nodes): rebuild relay-hop history on startup from path_json Kpa-clawbot/CoreScope#1643: under-attribute rather than mis-attribute).
   • Forward (from fromPubkey) and backward (from the observer) must not contradict each other. Define and test what happens when they meet or disagree.
2. Relay-role filter. Build the ingestor's prefix index from relay roles only (repeater, room). Match the server's definition and document it.
3. Start-up window. Prime the prefix index and graph before draining buffered ingest, or keep the buffer until they are primed.
4. Backfill. Periodically re-resolve observations within retention whose resolved_path is NULL (and optionally all-nil), in the ingestor (bug(db): vacuumOnStartup fails with SQLITE_BUSY when ingestor + server share DB (auto_vacuum migration #919 broken in single-container topology) Kpa-clawbot/CoreScope#1283 read/write split).
   • Work in bounded batches with a watermark, like the neighbor builder, rate-limited so live ingest is not starved of the SQLite write lock. It must be idempotent and logged.
   • Known consequence: the server indexes an observation once, when it first polls it. Backfilled values reach the server's indexes only on the next restart (Load). Document this, or propose how the server could pick up backfilled rows. The decision can be a follow-up.

Expected effect (estimate, from the #184 replay)

• About 85 % of today's NULL rows get resolved, with a range of 84.5–87.0 %. This relies on a proxy of the neighbor graph from /api/analytics/neighbor-graph, not neighbor_edges.
• The sum of traffic_share_score goes from about 14.8 (persisted today) to about 21.8. For comparison, the server's own resolver gives about 25.6.
• Disagreement between observers on the same (tx, prefix) is expected to be about 3.9 %. Part of it is real: two repeaters with the same prefix can both relay a flood.

Acceptance criteria

• Unit tests for observer-anchor resolution:
  • the last hop resolves via the observer;
  • the backward walk resolves earlier hops;
  • two neighbor candidates stay nil;
  • adverts are unchanged;
  • forward and backward agree.
  • Each test is red on master.
• A test that companions are never stored as relays.
• A test that rows ingested before the index is primed are resolved.
• Backfill tests: it is bounded, idempotent and resumable, and it does not block live ingest beyond a stated budget.
• Perf numbers: resolution cost per observation, and the backfill's write-lock hold time per batch.
• A replay or measurement report showing the new NULL share and the expected traffic-share sum.
• Staging verification (done by the lead/developer, not by the implementing agent): the sum settles near the estimate, does not grow with uptime, and spot-checked repeaters look plausible.
• Prod rollout only after staging, with a user-facing note explaining the score change.