Skip to content

development-runbook.md

Latest commit

 

History

History
305 lines (201 loc) · 10.1 KB

development-runbook.md

File metadata and controls

305 lines (201 loc) · 10.1 KB
Raw

Development Runbook

This runbook covers the local development workflow with Mosquitto, TimescaleDB 16, the MQTT subscriber/ingestor, and the synthetic publisher.

Requirements

The default workflow uses Docker Compose. The optional smoke-test script also uses the local Python environment from environment.yml.

Start The Four-Container Stack

Start the full local development stack:

docker compose -f examples/local-stack/docker-compose.yml up --build

This starts:

  • mqtt-publisher: publishes random traced payloads continuously from examples/local-stack/publisher-config.json.
  • mqtt-broker: local Mosquitto broker.
  • mqtt-subscriber: subscribes to the broker from examples/local-stack/subscriber-config.json and calls mqtt_ingest.ingest_message.
  • mqtt-subscriber-topics: subscribes to all broker topics from examples/local-stack/subscriber-topics-config.json and calls mqtt_ingest.ingest_topics.
  • timescaledb: local TimescaleDB 16 database.

The SQL bootstrap mounted into timescaledb comes from examples/sql/mqtt-ingest, which is now maintained as a Git submodule checkout.

The local database service uses the Timescale HA Docker image so timescaledb_toolkit is available during bootstrap.

The stack exposes these host ports for manual inspection:

  • MQTT broker: 127.0.0.1:1883
  • MQTT broker with Technitium DNS: mqtt.pi5.local:1883
  • TimescaleDB/Postgres protocol: 127.0.0.1:55432
  • Database: mqtt
  • User/password: postgres / postgres

The broker now runs with per-client MQTT credentials and topic ACLs sourced from examples/local-stack/mosquitto/, which is a public broker submodule checkout. Before first startup, populate examples/local-stack/mosquitto/passwords from passwords.example or switch that submodule to your private broker branch. The Compose stack keeps mqtt-broker as the internal Docker DNS name, while host and LAN clients can use mqtt.pi5.local once Technitium resolves that name to the Docker host IP.

The Compose service for the database is named timescaledb.

TimescaleDB starts with one generic ingest target: mqtt_ingest.messages. You do not need to create per-topic tables before subscribing.

View Logs

All services:

docker compose -f examples/local-stack/docker-compose.yml logs -f

Subscriber/ingestor only:

docker compose -f examples/local-stack/docker-compose.yml logs -f mqtt-subscriber

Publisher only:

docker compose -f examples/local-stack/docker-compose.yml logs -f mqtt-publisher

Broker only:

docker compose -f examples/local-stack/docker-compose.yml logs -f mqtt-broker

Query Stored Data

Inspect recent sensor rows, including parsed device_id and metric_name:

./scripts/dev/query-local-sensor-temp.sh

Compare publish, receive, and commit timing for the latest traced events:

./scripts/dev/query-local-trace-report.sh

Inspect 3-minute aggregate rows:

./scripts/dev/query-local-3m-aggregates.sh

Inspect 15-minute aggregate rows:

./scripts/dev/query-local-15m-aggregates.sh

Inspect 60-minute aggregate rows:

./scripts/dev/query-local-60m-aggregates.sh

Inspect 24-hour aggregate rows:

./scripts/dev/query-local-24h-aggregates.sh

Inspect power-energy reconciliation rows:

./scripts/dev/query-local-power-energy-reconciliation.sh

The aggregate queries include device_id, metric_name, plain in-bucket stats, percentile summaries such as median/p25/p75, trust metrics such as variance, standard error, and 95% confidence bounds, quality_score and its explainability fields, and the LOCF and linear boundary columns with their corresponding time-weighted averages.

Inspect the topic overview table:

./scripts/dev/query-local-topic-overview.sh

Direct database shell:

docker compose -f examples/local-stack/docker-compose.yml exec -T timescaledb \
  psql -U postgres -d mqtt

Stop And Reset

Stop the stack and remove volumes:

./scripts/dev/stop-local-test-stack.sh

Use this after SQL bootstrap changes so TimescaleDB reinitializes the tables and hypertables from the standalone SQL submodule at examples/sql/mqtt-ingest. The SQL bootstrap now also enables timescaledb_toolkit, so the Timescale image must include that extension.

Override Publisher Settings

For a step-by-step guide to adding new publishers, subscriber accounts, and ACLs before startup or while the broker is already running, see docs/secure-broker-howto.md.

The default mqtt-publisher service mounts examples/local-stack/publisher-config.json into the container at /config/publisher-config.json. The default mqtt-subscriber service mounts examples/local-stack/subscriber-config.json into the container at /config/subscriber-config.json. The topic-overview subscriber mounts examples/local-stack/subscriber-topics-config.json into the container at /config/subscriber-topics-config.json.

Edit that JSON file to change the publisher set, topics, or generator ranges without rewriting the Compose command. Edit the subscriber JSON file to change broker/database settings, MQTT credentials, or the topic filter list without rewriting the Compose command. Edit the topic-overview subscriber JSON file to change the overview ingest function or the topic filter scope without rewriting the Compose command. The default config includes both # and $SYS/# so broker status topics are recorded in mqtt_ingest.topic_overview.

Supported publisher topic generator kinds:

  • uniform
  • clipped_normal

For a temporary one-off publisher command, run:

docker compose -f examples/local-stack/docker-compose.yml run --rm mqtt-publisher \
  'exec python -m apps.publisher --topic sensors/node-2/temp --min-value 20 --max-value 30 --frequency-seconds 0.5'

The JSON or YAML format supports multiple publisher entries, and each publisher can define multiple topics with independent generators. Each topic generates its own value stream on every publish cycle.

Generate A Learned Twin Publisher Config

You can synthesize a fresh publisher-config.json style document from retained aggregate rows.

Example:

PYTHONPATH=src mqtt2postgres-twin-config \
  --db-host 127.0.0.1 \
  --db-port 55432 \
  --db-name mqtt \
  --db-user postgres \
  --db-password postgres \
  --topic-filter 'sensors/+/temp' \
  --output generated-publisher-config.json

Default behavior:

  • learns from mqtt_ingest.message_24h_aggregates
  • requires quality_status = 'rated'
  • requires quality_score >= 5.0
  • infers cadence from interval metrics when available
  • prefers clipped_normal generators and falls back to uniform when spread information is too weak

This is intended for feeding retained aggregate behavior back into the local MQTT simulation bench.

Manual Host Workflow

The four-container stack is the default development workflow. You can still run the ingestor and publisher directly on the host for debugging.

Create and activate the conda environment:

conda env create -f environment.yml
conda activate mqtt2postgres

Start only the local infrastructure:

docker compose -f examples/local-stack/docker-compose.yml up mqtt-broker timescaledb

Then run the ingestor locally in another terminal:

export POSTGRES_USERNAME=postgres
export POSTGRES_PASSWORD=postgres

python -m mqtt2postgres \
  --config path/to/subscriber.json

The ingestor is a long-running subscriber. Leave it running while publishing messages.

Publish five traced random values from the host:

python examples/publish_random.local.py \
  --host 127.0.0.1 \
  --port 1883 \
  --topic sensors/node-1/temp \
  --min-value 0 \
  --max-value 10 \
  --frequency-seconds 1 \
  --count 5 \
  --seed 7

Useful publisher options:

  • --count 5 publishes a fixed number of messages and exits.
  • --qos 1 changes MQTT publish QoS.
  • --seed 7 makes generated values repeatable.
  • --trace-id demo-trace-1 forces one shared trace id for the run.
  • --payload-format plain publishes raw numeric payloads instead of traced JSON.
  • --config path/to/publisher.json or --config path/to/publisher.yaml loads one publisher config file containing multiple publishers and multiple topics per publisher.
  • --config path/to/subscriber.json loads one JSON file for one subscriber with multiple topic filters.

Without --count, the publisher runs until stopped.

Smoke Test

Run the full smoke path:

./scripts/dev/run-local-smoke-test.sh

The smoke script is deterministic and separate from the continuous four-container workflow. It starts only mqtt-broker and timescaledb, runs the ingestor on the host, publishes traced messages for multiple devices under sensors/+/temp, and verifies rows in mqtt_ingest.messages, the four message_*_aggregates tables, and the four power_energy_*_reconciliation tables.

If your Python executable is not python3, override it:

PYTHON_BIN=python ./scripts/dev/run-local-smoke-test.sh

Run the secure auth/ACL smoke path:

./scripts/dev/run-secure-local-smoke-test.sh

This secure smoke path uses mqtt.pi5.local by default, validates authenticated publishers and subscribers, and checks that bad credentials and unauthorized publishes do not reach ingest storage.

Capture MQTT Traffic

Capture MQTT packets during a local run:

./scripts/dev/capture-mqtt-traffic.sh

The capture is written under .tmp/ as a .pcap file for tools such as Wireshark or tshark.

Troubleshooting

postgres is not running

The local database Compose service is named timescaledb. Scripts should use:

docker compose -f examples/local-stack/docker-compose.yml exec -T timescaledb ...

localhost:5432 connection refused from Docker

Inside an ingestor container, localhost points to the ingestor container. The four-container Compose stack uses service DNS names: mqtt-broker and timescaledb.

For standalone docker run against host-published ports, use host.docker.internal.

ModuleNotFoundError: No module named 'numpy'

Activate the conda environment before running the publisher or smoke test:

conda activate mqtt2postgres