feat: remove JSON filesystem storage

Author: SimmerV

POSTGRES.md

@@ -1,52 +1,43 @@
-# PostgreSQL Integration for MeshInfo
+# PostgreSQL Storage
 
-This document describes the PostgreSQL dual-write implementation for MeshInfo.
-
-## Overview
-
-MeshInfo now supports PostgreSQL as an alternative storage backend alongside JSON files. The implementation follows a **dual-write pattern** where:
-
-1. All writes go to **both** JSON files and PostgreSQL (when enabled)
-2. Reads can come from **either** JSON or PostgreSQL (configurable)
-3. JSON files remain authoritative during the transition period
-4. PostgreSQL failures never block JSON operations
+MeshInfo uses PostgreSQL as its storage backend. This document covers configuration, schema, and operational guidance.
 
 ## Configuration
 
 Add the following to your `config.toml`:
 
 ```toml
 [storage]
-read_from = "json"
-write_to = ["json", "postgres"]
+read_from = "postgres"
+write_to = ["postgres"]
 
 [storage.postgres]
-enabled = false
+enabled = true
 host = "postgres"
 port = 5432
 database = "meshinfo"
 username = "postgres"
-password = "password"
-min_pool_size = 5
-max_pool_size = 20
+password = "your_password"
+min_pool_size = 1
+max_pool_size = 5
 ```
 
 ### Configuration Options
 
-- **`read_from`**: `"json"` or `"postgres"` - Controls where data is loaded from on startup
-- **`write_to`**: Array of `["json", "postgres"]` - Controls which backends receive writes
-- **`postgres.enabled`**: `true` or `false` - Master switch for PostgreSQL functionality
-- **`postgres.host`**: Database server hostname
+- **`read_from`**: Must be `"postgres"`
+- **`write_to`**: Must be `["postgres"]`
+- **`postgres.enabled`**: Must be `true`
+- **`postgres.host`**: Database server hostname (use `"localhost"` if not using Docker)
 - **`postgres.port`**: Database server port (default: 5432)
 - **`postgres.database`**: Database name
 - **`postgres.username`**: Database user
 - **`postgres.password`**: Database password
-- **`postgres.min_pool_size`**: Minimum connection pool size (default: 5)
-- **`postgres.max_pool_size`**: Maximum connection pool size (default: 20)
+- **`postgres.min_pool_size`**: Minimum connection pool size (default: 1)
+- **`postgres.max_pool_size`**: Maximum connection pool size (default: 5)
 
 ## Database Schema
 
-The PostgreSQL schema stores data in a relational structure while maintaining compatibility with the JSON format. Key tables include:
+The schema is automatically created on first run. Key tables:
 
 - **`nodes`**: Core node information (ID, name, hardware, status)
 - **`node_positions`**: Historical position data
@@ -57,147 +48,48 @@ The PostgreSQL schema stores data in a relational structure while maintaining co
 - **`chat_messages`**: All chat messages
 - **`traceroutes`**: Complete traceroute history (JSONB payloads)
 
-The schema is automatically created on first run when PostgreSQL is enabled.
-
-## Migration
-
-To migrate existing JSON data to PostgreSQL:
-
-### Step 1: Enable PostgreSQL
-
-Update `config.toml`:
-```toml
-[storage.postgres]
-enabled = true
-```
-
-### Step 2: Start PostgreSQL
-
-If using Docker Compose, PostgreSQL should already be running. Otherwise, start it:
-
-```bash
-docker-compose up -d postgres
-```
-
-### Step 3: Run Migration Script
-
-```bash
-python scripts/migrate_json_to_postgres.py
-```
-
-The script will:
-- Connect to PostgreSQL
-- Create the database schema if needed
-- Import all nodes, chat messages, telemetry, and traceroutes from JSON files
-- Preserve all historical data
-- Report progress and any errors
-
-### Step 4: Enable Dual-Write
-
-Update `config.toml` to write to both backends:
-```toml
-[storage]
-read_from = "json"
-write_to = ["json", "postgres"]
-
-[storage.postgres]
-enabled = true
-```
-
-### Step 5: Verify Data Consistency
-
-- Monitor logs for any PostgreSQL write errors
-- Spot-check the API to ensure data is identical
-- Compare record counts between JSON and PostgreSQL
-
-### Step 6: Switch to PostgreSQL Reads
-
-Once confident in data consistency, switch to direct PostgreSQL queries:
-```toml
-[storage]
-read_from = "postgres"
-write_to = ["json", "postgres"]
-```
-
-**Important**: When `read_from: "postgres"`, the API queries PostgreSQL directly without loading data into memory. This provides:
-- Lower memory footprint
-- Always up-to-date data from the database
-- Better scalability for large datasets
-
 ## Architecture
 
 ### Write Flow
 
 1. MQTT message received
 2. Data stored in memory (MemoryDataStore) for internal use
 3. **Real-time write to PostgreSQL** (non-blocking, errors logged)
-4. Periodic write to JSON files (every 300 seconds by default)
 
-### Read Flow (JSON mode)
+### Read Flow
 
 1. Application starts
-2. Data loaded from JSON files into memory
-3. API serves from in-memory data structures
-4. Fast response times with full dataset in RAM
-
-### Read Flow (PostgreSQL mode)
-
-1. Application starts
-2. PostgreSQL connection established (no data loaded into memory)
-3. **API queries PostgreSQL directly** for each request
+2. PostgreSQL connection established
+3. API queries PostgreSQL directly for each request
 4. Lower memory footprint, always current data
-5. Efficient queries with proper indexing
 
 ### Error Handling
 
 - PostgreSQL write failures are logged but **never block** application execution
-- If PostgreSQL reads fail, the system automatically falls back to JSON
 - Connection pool handles transient network issues
 - Failed writes are logged for manual investigation
 
-## Data Retention
-
-- **JSON files**: Limited history (configurable via file-based rotation)
-- **PostgreSQL**: Unlimited history (all records preserved indefinitely)
-
 ## Performance Considerations
 
 ### Real-time Writes
 
-All writes to PostgreSQL happen in real-time as data arrives from MQTT. This ensures:
-- Minimal data loss in case of application crash
-- Up-to-date data in PostgreSQL at all times
-- No batch write delays
+All writes happen in real-time as data arrives from MQTT, ensuring minimal data loss on crash.
 
 ### Connection Pooling
 
-The implementation uses asyncpg connection pooling (5-20 connections by default) to handle concurrent writes efficiently.
+asyncpg connection pooling handles concurrent writes efficiently. Tune `min_pool_size` and `max_pool_size` based on your load.
 
 ### Indexing
 
-Key indexes are created on:
-- Node IDs
-- Timestamps
-- Foreign key relationships
-
-This ensures fast queries even with large datasets.
-
-## Backwards Compatibility
-
-The implementation fully supports:
-
-1. **Running without PostgreSQL**: Simply don't enable it in config
-2. **Downgrading to JSON-only**: Set `postgres.enabled: false` and restart
-3. **JSON as authoritative source**: Keep `read_from: "json"` during transition
+Key indexes are maintained on node IDs, timestamps, and foreign key relationships for fast queries even with large datasets.
 
 ## Monitoring
 
 PostgreSQL operations are logged at INFO and ERROR levels:
 
 ```
-INFO: PostgreSQL connection pool established
-INFO: Loaded 1234 nodes from PostgreSQL
-ERROR: Failed to write node abc123 to PostgreSQL: connection timeout
+INFO: PostgreSQL mode: Data will be queried directly from database
+ERROR: Failed to write node abc123 to Postgres: connection timeout
 ```
 
 Monitor these logs to ensure healthy operation.
@@ -206,54 +98,39 @@ Monitor these logs to ensure healthy operation.
 
 ### Connection Failures
 
-If PostgreSQL connection fails:
-1. Check that PostgreSQL container is running
-2. Verify connection settings in config.toml
-3. Check network connectivity
-4. Review PostgreSQL logs
+If PostgreSQL connection fails on startup, MeshInfo will log the error and exit. Check:
 
-The application will continue running with JSON-only mode.
+1. PostgreSQL container/service is running
+2. Connection settings in `config.toml` are correct (`host`, `port`, `username`, `password`)
+3. Network connectivity between MeshInfo and the database
 
 ### Data Inconsistencies
 
-To verify data consistency:
-
-1. **Count records**:
-   ```sql
-   SELECT COUNT(*) FROM nodes;
-   SELECT COUNT(*) FROM chat_messages;
-   SELECT COUNT(*) FROM telemetry;
-   SELECT COUNT(*) FROM traceroutes;
-   ```
-
-2. **Compare with JSON** via API endpoints
+To verify record counts:
 
-3. **Check for write errors** in application logs
+```sql
+SELECT COUNT(*) FROM nodes;
+SELECT COUNT(*) FROM chat_messages;
+SELECT COUNT(*) FROM telemetry;
+SELECT COUNT(*) FROM traceroutes;
+```
 
-### Migration Issues
+### Migrating from JSON Storage
 
-If migration fails:
-1. Check PostgreSQL logs for errors
-2. Verify JSON files are valid and readable
-3. Ensure sufficient disk space
-4. Try migrating data types individually (modify script)
+If you are upgrading from an older version that used JSON file storage, a one-time migration script is available:
 
-## Future Enhancements
+```bash
+docker exec -it meshinfo-meshinfo-1 python3 scripts/migrate_json_to_postgres.py
+```
 
-Potential future improvements:
-- Write-ahead log for failed PostgreSQL writes
-- Automatic retry logic with exponential backoff
-- Data validation and integrity checks
-- Performance metrics and monitoring
-- Support for read replicas
-- Automatic failover between backends
+This reads your existing JSON data files and imports them into PostgreSQL. Run it once before switching to PostgreSQL-only mode.
 
 ## Security
 
 - Use strong passwords for PostgreSQL
-- Consider using SSL/TLS for database connections in production
+- Consider SSL/TLS for database connections in production
 - Restrict database access via network policies
-- Regular backups of PostgreSQL data
+- Set up regular PostgreSQL backups (e.g., `pg_dump`)
 - Keep PostgreSQL updated with security patches
 
 ## Support

README.md

@@ -6,7 +6,7 @@ Realtime web UI to run against a Meshtastic regional or private mesh network.
 
 ## Overview
 
-MeshInfo is written in Python and connects to an MQTT server that is receiving Meshtastic messages for the purpose of visualizing and inspecting traffic. It (currently) uses a filesystem to persist content, such as node info and telemetry. There are plans to optionally support Postgres and SQLite3 as optional persistance storage methods.
+MeshInfo is written in Python and connects to an MQTT server that is receiving Meshtastic messages for the purpose of visualizing and inspecting traffic. It uses PostgreSQL to persist content such as node info and telemetry.
 
 To make deployment to run an instance for your mesh easy, Docker support is included. We recommend using Docker Compose with a personalized version of the `docker-compose.yml` file to most easily deploy it, but any seasoned Docker user can also use the Docker image alone.
 
@@ -72,7 +72,7 @@ cd meshinfo
 
 ##### Edit Configuration
 
-1. Copy and then edit the `config.toml.sample` to `config.toml` (or `config.json.sample` to `config.json` for legacy JSON format).
+1. Copy and then edit the `config.toml.sample` to `config.toml`.
 2. Copy `Caddyfile.sample` to `Caddyfile` then edit the `Caddyfile` and be sure it is setup for your hostname (FQDN if requiring Let's Encrypt cert to be generated) and your email address for the TLS line.
 
  - Caddy will request a cert of the FQDN, be sure to specify any subdomain. For example: `https://meshinfo.domain.com`.