feat(traceroute): pre-decode and store traceroute routes and hops on capture

[joroses]

TLDR

Decodes and stores incoming traceroute packets into dedicated database tables (traceroute_routes and traceroute_hops) as they arrive, rather than repeatedly re-parsing raw packet data on every page load. Also includes an optional CLI command to convert existing history. Prepares for PR 2 to enable fast, unlimited-window traceroute queries in the UI.

Summary

Currently, traceroute views and API endpoints query the raw packet_history table and decode payloads on the fly for every request.

To prevent browser timeouts, queries enforce artificial packet limits (e.g., LIMIT 20000 for graphs/nodes, or smaller fetch caps in /traceroute). This introduces significant issues:

• Truncated time windows: Scanning a fixed packet cap might only reach back a few hours—or as little as 30 minutes on /traceroute—rather than the requested 24-hour or 7-day window.
• Silent data loss: Legitimate RF links and routes outside that narrow slice are silently dropped from maps and graphs.
• High CPU/DB overhead: Repeating deserialization on 20,000+ records degrades overall database performance.

---

Solution (PR 1: Storage Layer)

This PR introduces dedicated storage for pre-decoded traceroutes and ordered radio hops during packet capture.

Note on Scope: Web pages and API endpoints continue querying raw packet history for now. Switching the views and queries over to the new tables will take place in PR 2.

New Tables & Indexes

Two derived tables pre-parse the payloads into indexed relational models:

Table	What it stores	Primary Key	Target Queries
traceroute_routes	One decoded route per traceroute packet, including full path JSON arrays, return path, and parse status	packet_id	Route lists, route patterns, and initiator/target participation
traceroute_hops	Individual point-to-point radio hops with hop order, direction, and measured SNR	(packet_id, direction, hop_index)	Link searches, topology graphs, map connections, and longest links

Indexes are created for timestamp, node (source/target), and link lookups ((from_node_id, to_node_id, timestamp)).

---

Migrating Existing History (Backfill CLI)

Backfilling is optional for deployment:

• Safe to start immediately: The service (web and capture) can be safely started with or without running the backfill. Newly captured traceroutes are stored in the new tables automatically from the moment the service starts.
• Why run it: Backfilling is only needed to adapt historical data for the upcoming reader migration in PR 2. If skipped, existing traceroutes simply remain in raw packet_history without populating the derived tables.

Web startup automatically creates the empty tables and indexes, but does not automatically backfill historical packets (to prevent database write-lock timeouts on large datasets during boot).

To prepare existing database history at your convenience, maintainers can run:

# Prepare existing history in resumable batches:
uv run malla-backfill-traceroutes --database /path/to/meshtastic_history.db

# Inspect readiness and record counts without modifying data:
uv run malla-backfill-traceroutes --database /path/to/meshtastic_history.db --check

---

Commits

• eb0cbb3 (Schema change, capture processing & backfill tool):
  • Creates traceroute_routes (route metadata, JSON paths, completion flags) and traceroute_hops (individual radio hops with directional SNR).
  • Updates live MQTT capture to decode and write traceroutes to these tables immediately in the same transaction.
  • Adds SQLite triggers to keep tables in sync if raw packets are inserted, modified, or deleted.
  • Adds the malla-backfill-traceroutes CLI tool to let maintainers migrate historical database records in safe, resumable batches.
• 95c6215 (Fix for timezone toggle input race):
  • Prevents url-filter-manager.js from overwriting text while a user is actively typing into filter inputs during asynchronous updates (resolving a flaky timezone toggle test on slower machines).

---

Verification

• Automated Tests: 30 unit/integration tests cover capture/backfill parity, malformed payloads, transaction rollbacks, duplicate gateway receptions, retention deletions, and parser version upgrades. Full E2E suite passes cleanly.
• Linting & Types: ruff and basedpyright pass cleanly.
• Tested on Real Data: Successfully backfilled 145,539 historical traceroutes and extracted 991,516 hops in ~75 seconds on real-world database history.