From 883aa88d48600ce695afac4bd7ec21e872be5c6b Mon Sep 17 00:00:00 2001 From: Justin Phelps Date: Tue, 19 Jul 2022 14:42:24 -0600 Subject: [PATCH 1/8] Fix markdown table. Document new DEBUG and MESSAGE_TYPE options. Docker Compose example. --- README.Docker.md | 66 +++++++++++++++++++++++++++++++++++------------- 1 file changed, 48 insertions(+), 18 deletions(-) diff --git a/README.Docker.md b/README.Docker.md index fba78fe..9cdb426 100644 --- a/README.Docker.md +++ b/README.Docker.md @@ -1,4 +1,4 @@ -# Using amirdm2mqtt in Docker. +# Using amirdm2mqtt in Docker If you want to run this under Docker you can do so. A Dockerfile has been provided so you can build your own container. @@ -6,41 +6,71 @@ If you want to run this under Docker you can do so. A Dockerfile has been provid Building should be a simple matter: - docker build -t amirdm2mqtt . +```shell +docker build -t amirdm2mqtt . +``` ## Configuration All configuration for the docker container is handled through environment variables. You can pass these to `docker run` using the -e flag. At a minimum you need to set `WATCHED_METERS`. | Environment Variable | Default | Required | Description | -|----------------------|----------|-------------| +|----------------------|---------|----------|-------------| | WATCHED_METERS | | Yes | A comma or space separated list of meters to watch | -| WH_MULTIPLIER | 1000 | No | multiplier to get reading to Watt Hours (Wh) | -| READINGS_PER_HOUR | 12 | No | number of IDM intervals per hour reported by the meter | +| WH_MULTIPLIER | `1000` | No | multiplier to get reading to Watt Hours (Wh) | +| READINGS_PER_HOUR | `12` | No | number of IDM intervals per hour reported by the meter | | MQTT_HOST | `127.0.0.1` | No | MQTT host to report to | -| MQTT_PORT | `1883' | No | MQTT port to use | +| MQTT_PORT | `1883` | No | MQTT port to use | | MQTT_USER | | No | MQTT username for authentication | | MQTT_PASSWORD | | No | MQTT password for authentication | +| MESSAGE_TYPE | `idm` | No | The message type rtlamr should output | +| DEBUG | `False` | No | Output debug messages | ## Running In order to run your container will need to be both privileged and have a volume mount to `/dev/bus/usb`. You can do that by adding these arguments to `docker run`: - --privileged -v /dev/bus/usb:/dev/bus/usb +```shell +--privileged -v /dev/bus/usb:/dev/bus/usb +``` You may also need to give it access to the network for your mqtt server. If you have not yet set one up you can do so with these commands: - docker network create --attachable mqtt - docker network connect mqtt +```shell +docker network create --attachable mqtt +docker network connect mqtt +``` A comman `docker run` command incorporating the above advice along with a common configuration is provided as an example: - docker run -it --name amridm2mqtt \ - --restart=unless-stopped \ - --network=mqtt \ - --privileged \ - -v /dev/bus/usb:/dev/bus/usb \ - -e WATCHED_METERS=12345678 \ - -e READINGS_PER_HOUR=4 \ - -e MQTT_HOST=mosquitto \ - amridm2mqtt +```shell +docker run -it --name amridm2mqtt \ + --restart=unless-stopped \ + --network=mqtt \ + --privileged \ + -v /dev/bus/usb:/dev/bus/usb \ + -e WATCHED_METERS=12345678 \ + -e READINGS_PER_HOUR=4 \ + -e MQTT_HOST=mosquitto \ + amridm2mqtt +``` + +## Docker Compose + +Here's an example using Docker Compose: + +```yaml + amridm2mqtt: + container_name: amridm2mqtt + build: amridm2mqtt + environment: + - WATCHED_METERS=12345678 + - READINGS_PER_HOUR=4 + - MQTT_HOST=mosquitto + privileged: true + devices: + - /dev/bus/usb:/dev/bus/usb + depends_on: + - mosquitto + restart: unless-stopped +``` From fde35c02020403b68626529e938ca20077814483 Mon Sep 17 00:00:00 2001 From: Justin Phelps Date: Tue, 19 Jul 2022 14:42:51 -0600 Subject: [PATCH 2/8] markdown lint formatting --- README.md | 14 +++++++------- 1 file changed, 7 insertions(+), 7 deletions(-) diff --git a/README.md b/README.md index 1e08369..ec5336f 100644 --- a/README.md +++ b/README.md @@ -1,12 +1,11 @@ # AMRIDM2MQTT: Send AMR/ERT Power Meter Data Over MQTT -##### Copyright (c) 2018 Ben Johnson. Distributed under MIT License. +## Copyright (c) 2018 Ben Johnson. Distributed under MIT License Using an [inexpensive rtl-sdr dongle](https://www.amazon.com/s/ref=nb_sb_noss?field-keywords=RTL2832U), it's possible to listen for signals from ERT compatible smart meters using rtlamr. This script runs as a daemon, launches rtl_tcp and rtlamr, and parses the output from rtlamr. If this matches your meter, it will push the data into MQTT for consumption by Home Assistant, OpenHAB, or custom scripts. TODO: Video for Home Assistant - ## Docker If you use Docker and would rather launch this under a container see . @@ -53,14 +52,13 @@ Install Go programming language & set gopath `sudo apt-get install golang` -https://github.com/golang/go/wiki/SettingGOPATH + If only running go to get rtlamr, just set environment temporarily with the following command `export GOPATH=$HOME/go` - -Install rtlamr https://github.com/bemasher/rtlamr +Install rtlamr `go get github.com/bemasher/rtlamr` @@ -71,6 +69,7 @@ To make things convenient, I'm copying rtlamr to /usr/local/bin ## Install ### Clone Repo + Clone repo into opt `cd /opt` @@ -110,7 +109,8 @@ Set amridm2mqtt to run on startup ### Configure Home Assistant To use these values in Home Assistant, -``` + +```yaml sensor: - platform: mqtt state_topic: "readings/12345678/meter_reading" @@ -121,7 +121,7 @@ sensor: state_topic: "readings/12345678/meter_rate" name: "Power Meter Avg Usage 5 mins" unit_of_measurement: W - ``` +``` ## Testing From aa9a5fc5504b3d5f67d38bf168ac407ecf4ddef1 Mon Sep 17 00:00:00 2001 From: Justin Phelps Date: Tue, 19 Jul 2022 14:43:14 -0600 Subject: [PATCH 3/8] add our DEBUG and MESSAGE_TYPE options to settings. --- settings_docker.py | 15 ++++++++++++++- settings_template.py | 12 ++++++++++++ 2 files changed, 26 insertions(+), 1 deletion(-) diff --git a/settings_docker.py b/settings_docker.py index 90fd525..7e227f2 100644 --- a/settings_docker.py +++ b/settings_docker.py @@ -7,7 +7,8 @@ for key in ['WATCHED_METERS']: if key not in os.environ: all_keys_found = False - print("Can't find key {0}, did you pass `-e {0}=` to `docker run`?".format(key)) + print( + "Can't find key {0}, did you pass `-e {0}=` to `docker run`?".format(key)) if not all_keys_found: print("\nPlease set the environment variables above.") @@ -56,3 +57,15 @@ # path to rtl_tcp RTL_TCP = '/usr/bin/rtl_tcp' + +# MESSAGE_TYPE we are looking for +# scm: Standard Consumption Message. Simple packet that reports total consumption. +# scm+: Similar to SCM, allows greater precision and longer meter ID's. +# idm: Interval Data Message. Provides differential consumption data for previous 47 intervals at 5 minutes per interval. +# netidm: Similar to IDM, except net meters (type 8) have different internal packet structure, number of intervals and precision. Also reports total power production. +# r900: Message type used by Neptune R900 transmitters, provides total consumption and leak flags. +# r900bcd: Some Neptune R900 meters report consumption as a binary-coded digits. +MESSAGE_TYPE = os.environ.get('MESSAGE_TYPE', 'idm') + +# DEBUG to output debug information +DEBUG = bool(os.environ.get('DEBUG', False)) diff --git a/settings_template.py b/settings_template.py index 1f2e21d..8101a8f 100644 --- a/settings_template.py +++ b/settings_template.py @@ -40,3 +40,15 @@ # path to rtl_tcp RTL_TCP = '/usr/bin/rtl_tcp' + +# MESSAGE_TYPE we are looking for +# scm: Standard Consumption Message. Simple packet that reports total consumption. +# scm+: Similar to SCM, allows greater precision and longer meter ID's. +# idm: Interval Data Message. Provides differential consumption data for previous 47 intervals at 5 minutes per interval. +# netidm: Similar to IDM, except net meters (type 8) have different internal packet structure, number of intervals and precision. Also reports total power production. +# r900: Message type used by Neptune R900 transmitters, provides total consumption and leak flags. +# r900bcd: Some Neptune R900 meters report consumption as a binary-coded digits. +MESSAGE_TYPE = 'idm' + +# DEBUG to output debug information +DEBUG = False From cecd2516cb5c51015b577aaf039f172bd73cce14 Mon Sep 17 00:00:00 2001 From: Justin Phelps Date: Tue, 19 Jul 2022 14:43:30 -0600 Subject: [PATCH 4/8] Field indexes for different MESSAGE_TYPES --- messagetypes.py | 22 ++++++++++++++++++++++ 1 file changed, 22 insertions(+) create mode 100644 messagetypes.py diff --git a/messagetypes.py b/messagetypes.py new file mode 100644 index 0000000..59a700b --- /dev/null +++ b/messagetypes.py @@ -0,0 +1,22 @@ +''' +MESSAGE_TYPE supported by rtlamr +scm: Standard Consumption Message. Simple packet that reports total consumption. +scm+: Similar to SCM, allows greater precision and longer meter ID's. +idm: Interval Data Message. Provides differential consumption data for previous 47 intervals at 5 minutes per interval. +netidm: Similar to IDM, except net meters (type 8) have different internal packet structure, number of intervals and precision. Also reports total power production. +r900: Message type used by Neptune R900 transmitters, provides total consumption and leak flags. +r900bcd: Some Neptune R900 meters report consumption as a binary-coded digits. + +These values define the field location for each reading, +when rtlamr is in CSV output. +''' + +IDM_FIELDS = 66 +IDM_METER_ID = 9 +IDM_CURRENT_READING = 15 +IDM_CURRENT_INTERVAL = 10 +IDM_MOST_RECENT_INTERVAL_USAGE = 16 + +SCM_FIELDS = 9 +SCM_METER_ID = 3 +SCM_CURRENT_READING = 7 From bf502d2d79ecf8f966c0c440416707e990b37ece Mon Sep 17 00:00:00 2001 From: Justin Phelps Date: Tue, 19 Jul 2022 14:43:48 -0600 Subject: [PATCH 5/8] Refactor to support multiple message types --- amridm2mqtt | 187 +++++++++++++++++++++++++++++++++------------------- 1 file changed, 119 insertions(+), 68 deletions(-) diff --git a/amridm2mqtt b/amridm2mqtt index 2e85146..cb7f3c2 100755 --- a/amridm2mqtt +++ b/amridm2mqtt @@ -14,11 +14,14 @@ import subprocess import signal import sys import time + import paho.mqtt.publish as publish import settings +import messagetypes + -# uses signal to shutdown and hard kill opened processes and self def shutdown(signum, frame): + '''uses signal to shutdown and hard kill opened processes and self''' rtltcp.send_signal(15) rtlamr.send_signal(15) time.sleep(1) @@ -26,88 +29,136 @@ def shutdown(signum, frame): rtlamr.send_signal(9) sys.exit(0) -signal.signal(signal.SIGTERM, shutdown) -signal.signal(signal.SIGINT, shutdown) - -# stores last interval id to avoid duplication, includes getter and setter -last_reading = {} - -auth = None - -if len(settings.MQTT_USER) and len(settings.MQTT_PASSWORD): - auth = {'username':settings.MQTT_USER, 'password':settings.MQTT_PASSWORD} - -DEBUG=os.environ.get('DEBUG', '').lower() in ['1', 'true', 't'] def debug_print(*args, **kwargs): if DEBUG: print(*args, **kwargs) + def get_last_interval(meter_id): return last_reading.get(meter_id, (None)) + def set_last_interval(meter_id, interval_ID): last_reading[meter_id] = (interval_ID) -# send data to MQTT broker defined in settings -def send_mqtt(topic, payload,): + +def send_mqtt(topic, payload): + '''send data to MQTT broker defined in settings''' try: - publish.single(topic, payload=payload, qos=1, hostname=settings.MQTT_HOST, port=settings.MQTT_PORT, auth=auth) + publish.single(topic, payload=payload, qos=1, + hostname=settings.MQTT_HOST, port=settings.MQTT_PORT, auth=auth) except Exception as ex: print("MQTT Publish Failed: " + str(ex)) -# start the rtl_tcp program -rtltcp = subprocess.Popen([settings.RTL_TCP + " > /dev/null 2>&1 &"], shell=True, - stdin=None, stdout=None, stderr=None, close_fds=True) -time.sleep(5) - -# start the rtlamr program. -rtlamr_cmd = [settings.RTLAMR, '-msgtype=idm', '-format=csv'] -rtlamr = subprocess.Popen(rtlamr_cmd, stdout=subprocess.PIPE, universal_newlines=True) - -while True: - try: - amrline = rtlamr.stdout.readline().strip() - flds = amrline.split(',') - - if len(flds) != 66: - # proper IDM results have 66 fields - continue - - # make sure the meter id is one we want - meter_id = int(flds[9]) - if settings.WATCHED_METERS and meter_id not in settings.WATCHED_METERS: - continue - # get some required info: current meter reading, current interval id, most recent interval usage - read_cur = int(flds[15]) - interval_cur = int(flds[10]) - idm_read_cur = int(flds[16]) - - # retreive the interval id of the last time we sent to MQTT +def send_meter_reading(reading, meter): + current_reading_in_kwh = (reading * settings.WH_MULTIPLIER) / 1000 + debug_print('Sending meter {} reading: {}'.format( + meter, current_reading_in_kwh)) + send_mqtt('readings/{}/meter_reading'.format(meter), + str(current_reading_in_kwh)) + + +def send_meter_usage(usage, meter): + rate = usage * settings.WH_MULTIPLIER * settings.READINGS_PER_HOUR + debug_print('Sending meter {} rate: {}'.format(meter, rate)) + send_mqtt('readings/{}/meter_rate'.format(meter), str(rate)) + + +def match_meterid(id): + if settings.WATCHED_METERS and id not in settings.WATCHED_METERS: + debug_print("meter id: ", id, + " doesn't match wanted meters: ", settings.WATCHED_METERS) + return False + return True + + +def parse_idm(flds): + # make sure the meter id is one we want + meter_id = int(flds[messagetypes.IDM_METER_ID]) + if match_meterid(meter_id): + # get some required info: + # current meter reading + # current interval id + # most recent interval usage + current_reading = int(flds[messagetypes.IDM_CURRENT_READING]) + current_interval = int(flds[messagetypes.IDM_CURRENT_INTERVAL]) + interval_usage = int(flds[messagetypes.IDM_MOST_RECENT_INTERVAL_USAGE]) + # retreive the interval id of the last time we sent data interval_last = get_last_interval(meter_id) - - if interval_cur != interval_last: - - # as observed on on my meter... - # using values set in settings... - # each idm interval is 5 minutes (12x per hour), - # measured in hundredths of a kilowatt hour - # take the last interval usage times 10 to get watt-hours, - # then times 12 to get average usage in watts - rate = idm_read_cur * settings.WH_MULTIPLIER * settings.READINGS_PER_HOUR - - current_reading_in_kwh = (read_cur * settings.WH_MULTIPLIER) / 1000 - - debug_print('Sending meter {} reading: {}'.format(meter_id, current_reading_in_kwh)) - send_mqtt('readings/{}/meter_reading'.format(meter_id), str(current_reading_in_kwh)) - - debug_print('Sending meter {} rate: {}'.format(meter_id, rate)) - send_mqtt('readings/{}/meter_rate'.format(meter_id), str(rate)) - + # if they don't match the current interval, send the data + if current_interval != interval_last: + send_meter_reading(current_reading, meter_id) + send_meter_usage(interval_usage, meter_id) # store interval ID to avoid duplicating data - set_last_interval(meter_id, interval_cur) - - except Exception as e: - debug_print('Exception squashed! {}: {}', e.__class__.__name__, e) - time.sleep(2) + set_last_interval(meter_id, current_interval) + + +def parse_scm(flds): + # make sure the meter id is one we want + meter_id = int(flds[messagetypes.SCM_METER_ID]) + if match_meterid(meter_id): + # get some required info: + # current meter reading + current_reading = int(flds[messagetypes.SCM_CURRENT_READING]) + # if they don't match the current interval, send the data + send_meter_reading(current_reading, meter_id) + + +if __name__ == "__main__": + + DEBUG = settings.DEBUG + + # Handle signals + signal.signal(signal.SIGTERM, shutdown) + signal.signal(signal.SIGINT, shutdown) + + # stores last interval id to avoid duplication, includes getter and setter + last_reading = {} + + # check and set our authentication + auth = None + if len(settings.MQTT_USER) and len(settings.MQTT_PASSWORD): + auth = {'username': settings.MQTT_USER, + 'password': settings.MQTT_PASSWORD} + + # start the rtl_tcp program + debug_print("Starting rtl_tcp...") + rtltcp = subprocess.Popen([settings.RTL_TCP + " > /dev/null 2>&1 &"], shell=True, + stdin=None, stdout=None, stderr=None, close_fds=True) + debug_print("Started rtl_tcp, waiting 5 seconds") + time.sleep(5) + + # start the rtlamr program + rtlamr_cmd = [settings.RTLAMR, + f'-msgtype={settings.MESSAGE_TYPE}', '-format=csv'] + debug_print("Starting rtlamr:", rtlamr_cmd) + rtlamr = subprocess.Popen( + rtlamr_cmd, stdout=subprocess.PIPE, universal_newlines=True) + + debug_print("Processing rtlamr output") + while True: + try: + # read a line from the process stdout + amrline = rtlamr.stdout.readline().strip() + flds = amrline.split(',') + debug_print(amrline) + + # Try to determine message type based on the number of fields + match len(flds): + case messagetypes.IDM_FIELDS: + debug_print( + "Number of fields suggests message type idm, parsing...") + parse_idm(flds) + case messagetypes.SCM_FIELDS: + debug_print( + "Number of fields suggests message type idm, parsing...") + parse_scm(flds) + case _: + debug_print("Unsupported number of fields: ", len(flds)) + continue + + except Exception as e: + debug_print('Exception squashed! {}: {}', e.__class__.__name__, e) + time.sleep(2) From 6a5ae5f211eddb936019cc8b93912a5bc0f9ab7c Mon Sep 17 00:00:00 2001 From: Justin Phelps Date: Tue, 19 Jul 2022 14:49:19 -0600 Subject: [PATCH 6/8] Output our python version during the docker build --- Dockerfile | 1 + 1 file changed, 1 insertion(+) diff --git a/Dockerfile b/Dockerfile index b08b039..e38ea33 100644 --- a/Dockerfile +++ b/Dockerfile @@ -13,6 +13,7 @@ RUN apt update && \ rtl-sdr && \ rm -rf /var/lib/apt/lists/* RUN go get github.com/bemasher/rtlamr +RUN python3 -V # Copy files into place COPY * /amridm2mqtt/ From 777a9ecbafd9244b3b13beeb4f53adfaecc3b815 Mon Sep 17 00:00:00 2001 From: Justin Phelps Date: Tue, 19 Jul 2022 14:49:40 -0600 Subject: [PATCH 7/8] Python < 3.10 in the container, so match isn't supported --- amridm2mqtt | 22 ++++++++++------------ 1 file changed, 10 insertions(+), 12 deletions(-) diff --git a/amridm2mqtt b/amridm2mqtt index cb7f3c2..2c93121 100755 --- a/amridm2mqtt +++ b/amridm2mqtt @@ -146,18 +146,16 @@ if __name__ == "__main__": debug_print(amrline) # Try to determine message type based on the number of fields - match len(flds): - case messagetypes.IDM_FIELDS: - debug_print( - "Number of fields suggests message type idm, parsing...") - parse_idm(flds) - case messagetypes.SCM_FIELDS: - debug_print( - "Number of fields suggests message type idm, parsing...") - parse_scm(flds) - case _: - debug_print("Unsupported number of fields: ", len(flds)) - continue + field_count = len(flds) + if field_count == messagetypes.IDM_FIELDS: + debug_print("Number of fields suggests message type idm, parsing...") + parse_idm(flds) + elif field_count == messagetypes.SCM_FIELDS: + debug_print("Number of fields suggests message type idm, parsing...") + parse_scm(flds) + else: + debug_print("Unsupported number of fields: ", field_count) + continue except Exception as e: debug_print('Exception squashed! {}: {}', e.__class__.__name__, e) From 200c706f666d8947aa06aad82dee1b5ca539bfaf Mon Sep 17 00:00:00 2001 From: Justin Phelps Date: Tue, 19 Jul 2022 14:50:46 -0600 Subject: [PATCH 8/8] Typo in the name of the tool --- README.Docker.md | 4 ++-- 1 file changed, 2 insertions(+), 2 deletions(-) diff --git a/README.Docker.md b/README.Docker.md index 9cdb426..6879c81 100644 --- a/README.Docker.md +++ b/README.Docker.md @@ -1,4 +1,4 @@ -# Using amirdm2mqtt in Docker +# Using amridm2mqtt in Docker If you want to run this under Docker you can do so. A Dockerfile has been provided so you can build your own container. @@ -7,7 +7,7 @@ If you want to run this under Docker you can do so. A Dockerfile has been provid Building should be a simple matter: ```shell -docker build -t amirdm2mqtt . +docker build -t amridm2mqtt . ``` ## Configuration