Make solismon resilient: never crash, retry until recovered - #1
Merged
Conversation
The monitor could be permanently killed by routine, transient conditions (inverter briefly unreachable, flaky Wi-Fi stick, MQTT broker down) and would slowly leak sockets/threads on a long-running host. This reworks the app so any error is logged and the process keeps running, actively retrying until it recovers and then continuing its work. main.py: - Remove all exit()/SystemExit paths; scrape_solis() now logs and returns True/False instead of terminating. Bounded read retries return False (no exit, no hot-loop, no stale/unbound regs consumption). - Run scraping in a background daemon thread (scrape_loop) that retries with exponential backoff capped at CHECK_INTERVAL on failure and resets to the normal cadence on success, so it recovers fast and never hangs or hammers. - Guarantee Modbus connection teardown in a finally (fixes fd/thread leak) and always disconnect the MQTT client. - Build metrics into a local dict and publish atomically on success, so a failed cycle keeps the last good snapshot (and avoids a dict-resize race). - Prometheus collect() now only serves the latest snapshot (instant, never blocks/raises); the exporter start retries a port-bind failure instead of exiting. - Guard timestamp parsing and register indexing against malformed frames. - Install SIGTERM/SIGINT handlers for clean, logged shutdown (Docker/Home Assistant stop with SIGTERM, which previously killed it with no cleanup). config/config.py: - Make config parsing total: a bad env var / option logs a warning and falls back to the default instead of raising at import time. - Clamp CHECK_INTERVAL to >= 1 to prevent a negative value crashing sleep() and a zero value busy-looping. Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Summary
Makes the monitor never crash. Any error is now logged and the process keeps running, actively retrying until it recovers and then continuing its work. Previously, routine transient conditions permanently killed the daemon.
Why
scrape_solis()calledexit(1)on a connect failure or after 3 failed reads. Becausefrom sys import exitraisesSystemExit(aBaseException), it bypassed everyexcept Exceptionand killed the process — so the inverter being briefly unreachable (overnight, Wi-Fi blip, reboot) permanently stopped monitoring.exit(1)'d, which under Home Assistantboot: autobecomes a restart/crash-loop.SIGTERM(what Docker/HA send to stop) killed the process at the C level with no clean-shutdown path.Changes
main.pyexit()/SystemExitpaths;scrape_solis()logs and returnsTrue/False. Bounded read retriesreturn False(no exit, no hot-loop, no stale/unboundregs).scrape_loop) that retries with exponential backoff capped atCHECK_INTERVALon failure and resets to normal cadence on success — recovers fast, never hangs or hammers, and can never die.finally(fixes the leak) and always disconnect the MQTT client.collect()now only serves the latest snapshot (instant, never blocks/raises); exporter startup retries a port-bind failure instead of exiting.SIGTERM/SIGINThandlers for clean, logged shutdown.config/config.pyCHECK_INTERVALto>= 1(preventssleep(-30)crashing and0busy-looping).Behavior change
Prometheus mode now polls the inverter every
CHECK_INTERVALin a background thread (and/metricsserves the latest snapshot instantly), instead of polling only when scraped. This is what enables retry-until-recovered without hanging HTTP requests.Verification
main.pywith a mock inverter that fails twice then recovers: it retried, recovered, kept publishing, and onSIGTERMexited 0 with no traceback — in both MQTT and Prometheus modes.🤖 Generated with Claude Code