diff --git a/README.md b/README.md index 5ba0b3c46..b9c5054be 100644 --- a/README.md +++ b/README.md @@ -1,273 +1,224 @@ -# CoreScope +# CoreScope for MeshView -[![Go Server Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/master/.badges/go-server-coverage.json)](https://github.com/Kpa-clawbot/CoreScope/actions/workflows/deploy.yml) -[![Go Ingestor Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/master/.badges/go-ingestor-coverage.json)](https://github.com/Kpa-clawbot/CoreScope/actions/workflows/deploy.yml) -[![E2E Tests](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/master/.badges/e2e-tests.json)](https://github.com/Kpa-clawbot/CoreScope/actions/workflows/deploy.yml) -[![Frontend Coverage](https://img.shields.io/endpoint?url=https://raw.githubusercontent.com/Kpa-clawbot/CoreScope/master/.badges/frontend-coverage.json)](https://github.com/Kpa-clawbot/CoreScope/actions/workflows/deploy.yml) -[![Deploy](https://github.com/Kpa-clawbot/CoreScope/actions/workflows/deploy.yml/badge.svg)](https://github.com/Kpa-clawbot/CoreScope/actions/workflows/deploy.yml) +[![CI](https://github.com/dborup/CoreScope/actions/workflows/deploy.yml/badge.svg?branch=master)](https://github.com/dborup/CoreScope/actions/workflows/deploy.yml) -> High-performance mesh network analyzer powered by Go. Sub-millisecond packet queries, ~300 MB memory for 56K+ packets, real-time WebSocket broadcast, full channel decryption. +This repository contains the CoreScope fork used by MeshView, a community service for exploring the Danish MeshCore network. It builds on the [upstream CoreScope project](https://github.com/Kpa-clawbot/CoreScope) with changes maintained for MeshView. -Self-hosted, open-source MeshCore packet analyzer. Collects MeshCore packets via MQTT, decodes them in real time, and presents a full web UI with live packet feed, interactive maps, channel chat, packet tracing, and per-node analytics. +**Explore MeshView:** [meshview.dk](https://meshview.dk) · [API documentation](https://meshview.dk/api/docs) · [OpenAPI specification](https://meshview.dk/api/spec) -## ⚡ Performance +CoreScope collects MeshCore packets from MQTT brokers, decodes them, and presents a web interface with live packet feeds, maps, channel messages, packet tracing, and node analytics. MeshView is the hosted service; this repository contains the software you can also run on your own infrastructure. -The Go backend serves all 40+ API endpoints from an in-memory packet store with 5 indexes (hash, txID, obsID, observer, node). SQLite is for persistence only — reads never touch disk. +## Features -| Metric | Value | -|--------|-------| -| Packet queries | **< 1 ms** (in-memory) | -| All API endpoints | **< 100 ms** | -| Memory (56K packets) | **~300 MB** (vs 1.3 GB on Node.js) | -| WebSocket broadcast | **Real-time** to all connected browsers | -| Channel decryption | **AES-128-ECB** with rainbow table | -| GOMEMLIMIT (memory-constrained hosts) | **set to ≥1.5× working set** (e.g. 1536 MiB on a 2 GB Pi for a ~1 GB store). Lower values trigger a GC death-spiral. Configure via the `GOMEMLIMIT` env var or `runtime.maxMemoryMB` in `config.json`; env wins. Applies to both server and ingestor. See [#1010](https://github.com/Kpa-clawbot/CoreScope/issues/1010). | +The screenshots below are examples inherited from upstream CoreScope. They illustrate the software's features and may differ from the current MeshView interface and data. -See [PERFORMANCE.md](PERFORMANCE.md) for full benchmarks. +### Live Trace Map -## ✨ Features +Watch packet routes on an animated map, or use the VCR controls to replay recorded activity and move through the timeline. -### 📡 Live Trace Map -Real-time animated map with packet route visualization, VCR-style playback controls, and a retro LCD clock. Replay the last 24 hours of mesh activity, scrub through the timeline, or watch packets flow live at up to 4× speed. +![Upstream CoreScope example: live VCR playback](docs/screenshots/MeshVCR.gif) -![Live VCR playback — watch packets flow across the Bay Area mesh](docs/screenshots/MeshVCR.gif) +### Packet Feed -### 📦 Packet Feed -Filterable real-time packet stream with byte-level breakdown, Excel-like resizable columns, and a detail pane. Toggle "My Nodes" to focus on your mesh. +Filter the packet stream, inspect individual bytes, resize table columns, and focus on your own nodes. -![Packets view](docs/screenshots/packets1.png) +![Upstream CoreScope example: packets view](docs/screenshots/packets1.png) -### 🗺️ Network Overview -At-a-glance mesh stats — node counts, packet volume, observer coverage. +### Network Overview -![Network overview](docs/screenshots/mesh-overview.png) +Explore node counts, packet volume, and observer coverage. -### 📊 Node Analytics -Per-node deep dive with interactive charts: activity timeline, packet type breakdown, SNR distribution, hop count analysis, peer network graph, and hourly heatmap. +![Upstream CoreScope example: network overview](docs/screenshots/mesh-overview.png) -![Node analytics](docs/screenshots/node-analytics.png) +### Node Analytics -### 💬 Channel Chat -Decoded group messages with sender names, @mentions, timestamps — like reading a Discord channel for your mesh. +Inspect a node's activity, packet types, SNR, hop counts, neighboring nodes, and activity over time. -![Channels](docs/screenshots/channels1.png) +![Upstream CoreScope example: node analytics](docs/screenshots/node-analytics.png) -### 📱 Mobile Ready -Full experience on your phone — proper touch controls, iOS safe area support, and a compact VCR bar. +### Channel Messages -Live view on iOS +Read decoded group messages when the corresponding channel keys are available. Hashtag channel keys can be derived automatically. -### And More +![Upstream CoreScope example: channels](docs/screenshots/channels1.png) -- **11 Analytics Tabs** — RF, topology, channels, hash stats, distance, route patterns, and more -- **Node Directory** — searchable list with role tabs, detail panel, QR codes, advert timeline -- **Packet Tracing** — follow individual packets across observers with SNR/RSSI timeline -- **Observer Status** — health monitoring, packet counts, uptime, per-observer analytics -- **Hash Collision Matrix** — detect address collisions across the mesh -- **Channel Key Auto-Derivation** — hashtag channels (`#channel`) keys derived via SHA256 -- **Multi-Broker MQTT** — connect to multiple brokers with per-source IATA filtering -- **Dark / Light Mode** — auto-detects system preference, map tiles swap too -- **Theme Customizer** — design your theme in-browser, export as `theme.json` -- **Global Search** — search packets, nodes, and channels (Ctrl+K) -- **Shareable URLs** — deep links to packets, channels, and observer detail pages -- **Protobuf API Contract** — typed API definitions in `proto/` -- **Accessible** — ARIA patterns, keyboard navigation, screen reader support +### Mobile Interface -## Quick Start +Use touch controls, mobile navigation, and the compact playback controls on a phone. -### Pre-built Image (Recommended) +Upstream CoreScope example: live view on iOS -No build step required — just run: +### More Tools -```bash -docker run -d --name corescope \ - --restart=unless-stopped \ - -p 80:80 -p 1883:1883 \ - -v /your/data:/app/data \ - ghcr.io/kpa-clawbot/corescope:latest -``` - -Open `http://localhost` — done. No config file needed; CoreScope starts with sensible defaults. +- **Node directory** — search nodes, inspect their roles, view adverts, and open node details. +- **Packet tracing** — follow packets across observers with SNR and RSSI measurements. +- **Observer status** — inspect health, packet counts, and observer analytics. +- **Network analytics** — explore RF activity, topology, hash collisions, distance, routes, and scopes. +- **Multiple MQTT sources** — collect data from several brokers with source-specific region filters. +- **Themes and customization** — choose light or dark mode, adjust the interface, and export theme settings. +- **Shareable views** — link directly to nodes, packets, channels, and filtered views. -For HTTPS with a custom domain, add `-p 443:443` and mount your Caddyfile: -```bash -docker run -d --name corescope \ - --restart=unless-stopped \ - -p 80:80 -p 443:443 -p 1883:1883 \ - -v /your/data:/app/data \ - -v /your/Caddyfile:/etc/caddy/Caddyfile:ro \ - -v /your/caddy-data:/data/caddy \ - ghcr.io/kpa-clawbot/corescope:latest -``` +## Run Your Own Instance -Disable built-in services with `-e DISABLE_MOSQUITTO=true` or `-e DISABLE_CADDY=true`, or drop a `.env` file in your data volume. See [docs/deployment.md](docs/deployment.md) for the full reference. +### Build This Fork -### Build from Source +You need Git and Docker. Build the image from this repository to run the MeshView fork: ```bash -git clone https://github.com/Kpa-clawbot/CoreScope.git +git clone --branch master https://github.com/dborup/CoreScope.git cd CoreScope -./manage.sh setup +docker build -t corescope-meshview:local . ``` -The setup wizard walks you through config, domain, HTTPS, build, and run. +The `corescope-meshview:local` tag is created on your machine. The published `ghcr.io/kpa-clawbot/corescope` images belong to upstream CoreScope. -```bash -./manage.sh status # Health check + packet/node counts -./manage.sh logs # Follow logs -./manage.sh backup # Backup database -./manage.sh update # Pull latest + rebuild + restart -./manage.sh mqtt-test # Check if observer data is flowing -./manage.sh help # All commands -``` +### Configure a Local Instance -### Configure +The container can start with its built-in defaults. To configure the MQTT source explicitly, create a data directory and save the following as `data/config.json`. This example uses the broker included in the container and contains no MeshView deployment settings. -Copy `config.example.json` to `config.json` and edit: +```bash +mkdir -p data +``` ```json { - "port": 3000, - "mqtt": { - "broker": "mqtt://localhost:1883", - "topic": "meshcore/+/+/packets" - }, "mqttSources": [ { - "name": "remote-feed", - "broker": "mqtts://remote-broker:8883", - "topics": ["meshcore/+/+/packets"], - "username": "user", - "password": "pass", - "iataFilter": ["SJC", "SFO", "OAK"] + "name": "local", + "broker": "mqtt://localhost:1883", + "topics": ["meshcore/#"] } - ], - "channelKeys": { - "public": "8b3387e9c5cdea6ac9e5edbaa115cd72" - }, - "defaultRegion": "SJC" + ] } ``` -| Field | Description | -|-------|-------------| -| `port` | HTTP server port (default: 3000) | -| `mqtt.broker` | Local MQTT broker URL (`""` to disable) | -| `mqttSources` | External MQTT broker connections (optional) | -| `channelKeys` | Channel decryption keys (hex). Hashtag channels auto-derived via SHA256 | -| `defaultRegion` | Default IATA region code for the UI | -| `dbPath` | SQLite database path (default: `data/meshcore.db`) | +Start the container from the repository root: -### Environment Variables +```bash +docker run -d --name corescope-meshview \ + --restart=unless-stopped \ + -p 127.0.0.1:8080:80 \ + -p 127.0.0.1:1883:1883 \ + --mount "type=bind,source=$(pwd)/data,target=/app/data" \ + corescope-meshview:local +``` -| Variable | Description | -|----------|-------------| -| `PORT` | Override config port | -| `DB_PATH` | Override SQLite database path | +Open [http://localhost:8080](http://localhost:8080). The included Caddy proxy forwards HTTP requests to the Go server. The database and configuration persist in `./data`; packets appear once an observer publishes to the local MQTT broker. -## Architecture +This example exposes HTTP and MQTT on the Docker host's loopback interface. A publisher on that host can connect to `mqtt://localhost:1883`. To collect from an external broker, replace `mqttSources` with that broker's connection settings and restart the container. You can then disable the included broker with `-e DISABLE_MOSQUITTO=true` and omit the MQTT port mapping. +```bash +docker logs -f corescope-meshview +docker restart corescope-meshview ``` - ┌─────────────────────────────────────────────┐ - │ Docker Container │ - │ │ -Observer → USB → │ Mosquitto ──→ Go Ingestor ──→ SQLite DB │ - meshcoretomqtt → MQTT ──→│ │ │ - │ Go HTTP Server ──→ WebSocket │ - │ │ │ │ - │ Caddy (HTTPS) ←───────┘ │ - └────────────────────┼────────────────────────┘ - │ - Browser + +For a public deployment, configure your domain and HTTPS using Caddy or your own reverse proxy. The included broker allows anonymous connections; configure broker authentication before exposing it beyond the host. See [deployment documentation](docs/deployment.md) for the available components and options, using the locally built image above for this fork. + +### Configuration Reference + +The container reads `config.json` from the directory mounted at `/app/data`. [config.example.json](config.example.json) documents additional settings, including branding, map defaults, areas, channel keys, filters, and retention. Adapt individual settings to your own network: that file also contains upstream example brokers and geographic filters. + +| Setting | Purpose | +|---------|---------| +| `mqttSources` | MQTT brokers, subscriptions, credentials, and optional `iataFilter` values. | +| `channelKeys` | Keys for channels you want to decode. Hashtag channel keys can be derived automatically. | +| `branding` | Instance name, tagline, logo, and related links. | +| `packetStore.maxMemoryMB` | Estimated in-memory packet-store budget; total process memory also includes other allocations. | +| `packetStore.retentionHours` | Packet history retained in memory. | +| `retention.packetDays` | Packet retention in SQLite, managed by the ingestor. | +| `apiKey` | Key for protected administration endpoints. Omit it to leave those endpoints disabled; use a unique strong key if you enable them. | + +The standard container starts the server on internal port `3000` with database `/app/data/meshcore.db`; change the host-facing Docker port mapping to choose a different external port. If you set `DISABLE_CADDY=true`, publish internal port `3000` instead of `80` and let your own reverse proxy handle HTTP/HTTPS. + +When running the Go binaries directly, use the server's `-port` and `-db` flags for explicit overrides. The server uses `DB_PATH` only when `dbPath` is absent from its configuration; `PORT` is not a server configuration override. `GOMEMLIMIT` and the `runtime` settings control the Go runtime's soft memory limit separately from the packet-store budget. + +## Architecture + +```text +MeshCore observers + │ + ▼ +MQTT broker(s) ──► Go ingestor ──► SQLite database + │ + ▼ + Go HTTP server + ┌────────┴────────┐ + │ │ + REST + static UI WebSocket + │ │ + └────────┬────────┘ + ▼ + Caddy / reverse proxy + │ + ▼ + Browser ``` -**Two-process model:** The Go ingestor handles MQTT ingestion and packet decoding. The Go HTTP server loads all packets into an in-memory store on startup (5 indexes for fast lookups) and serves the REST API + WebSocket broadcast. Both are managed by supervisord inside a single container with Caddy for HTTPS and Mosquitto for local MQTT. +The ingestor owns packet ingestion, decoding, database migrations, and retention. The HTTP server opens SQLite read-only, maintains an indexed packet store with configurable memory and time limits, and polls for new data to broadcast over WebSocket. Responses use a combination of in-memory data, cached analytics, and SQLite queries; older packet windows can fall back to SQLite. + +The frontend is plain HTML, CSS, and JavaScript served directly by Go, with no frontend build step. The Docker image runs the Go server and ingestor under supervisord and includes Mosquitto and Caddy as optional services. -## MQTT Setup +Memory use and query latency depend on the dataset, enabled analytics, retention settings, and host. Tune the packet-store and runtime memory settings for your deployment rather than treating historical benchmark figures as guarantees for MeshView or other instances. -1. **Flash an observer node** with `MESH_PACKET_LOGGING=1` build flag -2. **Connect via USB** to a host running [meshcoretomqtt](https://github.com/Cisien/meshcoretomqtt) -3. **Configure meshcoretomqtt** with your IATA region code and MQTT broker address -4. **Packets appear** on topic `meshcore/{IATA}/{PUBKEY}/packets` +## Connect Observers -Or POST raw hex packets to `POST /api/packets` for manual injection. +Configure a MeshCore observer and a compatible MQTT publisher, such as [meshcoretomqtt](https://github.com/Cisien/meshcoretomqtt), following that publisher's setup instructions. Point it at your broker and choose the appropriate IATA region code. Packet topics use the form `meshcore/{IATA}/{PUBKEY}/packets`; the `meshcore/#` subscription in the example above also receives related status topics. ## Project Structure -``` -corescope/ +```text +CoreScope/ ├── cmd/ -│ ├── server/ # Go HTTP server + WebSocket + REST API -│ │ ├── main.go # Entry point -│ │ ├── routes.go # 40+ API endpoint handlers -│ │ ├── store.go # In-memory packet store (5 indexes) -│ │ ├── db.go # SQLite persistence layer -│ │ ├── decoder.go # MeshCore packet decoder -│ │ ├── websocket.go # WebSocket broadcast -│ │ └── *_test.go # 327 test functions -│ └── ingestor/ # Go MQTT ingestor -│ ├── main.go # MQTT subscription + packet processing -│ ├── decoder.go # Packet decoder (shared logic) -│ ├── db.go # SQLite write path -│ └── *_test.go # 53 test functions -├── proto/ # Protobuf API definitions -├── public/ # Vanilla JS frontend (no build step) -│ ├── index.html # SPA shell -│ ├── app.js # Router, WebSocket, utilities -│ ├── packets.js # Packet feed + hex breakdown -│ ├── map.js # Leaflet map + route visualization -│ ├── live.js # Live trace + VCR playback -│ ├── channels.js # Channel chat -│ ├── nodes.js # Node directory + detail views -│ ├── analytics.js # 11-tab analytics dashboard -│ └── style.css # CSS variable theming (light/dark) -├── docker/ -│ ├── supervisord-go.conf # Process manager (server + ingestor) -│ ├── mosquitto.conf # MQTT broker config -│ ├── Caddyfile # Reverse proxy + HTTPS -│ └── entrypoint-go.sh # Container entrypoint -├── Dockerfile # Multi-stage Go build + Alpine runtime -├── config.example.json # Example configuration -├── test-*.js # Node.js test suite (frontend + legacy) -└── tools/ # Generators, E2E tests, utilities +│ ├── server/ # Read-only SQLite access, REST API, WebSocket, static UI +│ ├── ingestor/ # MQTT ingestion, decoding, SQLite writes and migrations +│ ├── decrypt/ # Channel decryption CLI +│ └── migrate/ # Database migration CLI, also used for test fixtures +├── internal/ # Shared Go packages +├── public/ # HTML, CSS, and vanilla JavaScript +├── proto/ # Protocol buffer definitions +├── docker/ # Supervisor, Mosquitto, Caddy, and entrypoint configuration +├── Dockerfile # Go build and Alpine runtime image +├── config.example.json # Configuration reference with upstream examples +├── test-fixtures/ # Database fixtures for local and CI browser tests +├── test-*.js # Frontend unit and browser tests +├── scripts/ # Test, coverage, and maintenance scripts +└── tools/ # Development utilities ``` -## For Developers - -### Test Suite +## Development and Tests -**380 Go tests** covering the backend, plus **150+ Node.js tests** for the frontend and legacy logic, plus **49 Playwright E2E tests** for browser validation. +Use a Go toolchain compatible with the module files, plus Node.js and npm for the frontend tests. Run these commands from the repository root: ```bash -# Go backend tests -cd cmd/server && go test ./... -v -cd cmd/ingestor && go test ./... -v +# Go backend tests; each command leaves the shell in the repository root. +(cd cmd/server && go test ./...) +(cd cmd/ingestor && go test ./...) -# Node.js frontend + integration tests +# Frontend test dependencies and tests. +npm ci +npm run test:unit npm test - -# Playwright E2E (requires running server on localhost:3000) -node test-e2e-playwright.js ``` -### Generate Test Data +`npm test` runs the JavaScript suites listed in [test-all.sh](test-all.sh). Go tests run separately. + +Browser tests need Chromium and a running local Go server with a populated test database. For the fixture preparation, migrations, and server startup used by CI, see [.github/workflows/deploy.yml](.github/workflows/deploy.yml). Run browser tests against a local test instance, not the public MeshView service: ```bash -node tools/generate-packets.js --api --count 200 +npx playwright install chromium +BASE_URL=http://localhost:3000 node test-e2e-playwright.js ``` -### Migrating from Node.js - -If you're running an existing Node.js deployment, see [docs/go-migration.md](docs/go-migration.md) for a step-by-step guide. The Go engine reads the same SQLite database and `config.json` — no data migration needed. +Set `BASE_URL` to your local test server's port, or set `CHROMIUM_PATH` if using an existing Chromium installation. The CI workflow runs additional browser suites beyond this entry point. -## Contributing +The [CI workflow for this fork](https://github.com/dborup/CoreScope/actions/workflows/deploy.yml) runs tests and builds the container image locally in CI. Registry publishing, release uploads, deployment, and badge-file commits are restricted to the upstream repository by the workflow guards. -Contributions welcome. Please read [AGENTS.md](AGENTS.md) for coding conventions, testing requirements, and engineering principles before submitting a PR. +## Contributing and Upstream -**Live instance:** [analyzer.00id.net](https://analyzer.00id.net) — all API endpoints are public, no auth required. +Open issues and pull requests in [dborup/CoreScope](https://github.com/dborup/CoreScope). Read [AGENTS.md](AGENTS.md) for the project's development conventions and validation requirements. -**API Documentation:** CoreScope auto-generates an OpenAPI 3.0 spec. Browse the interactive Swagger UI at [`/api/docs`](https://analyzer.00id.net/api/docs) or fetch the machine-readable spec at [`/api/spec`](https://analyzer.00id.net/api/spec). +CoreScope was developed in the [upstream Kpa-clawbot/CoreScope project](https://github.com/Kpa-clawbot/CoreScope). This fork retains that history and attribution while maintaining changes for MeshView. MeshCore firmware is developed separately at [meshcore-dev/MeshCore](https://github.com/meshcore-dev/MeshCore). ## License -GPL-3.0-or-later +[GPL-3.0-or-later](LICENSE).