Skip to content

Make solismon resilient: never crash, retry until recovered - #1

Merged
dnviti merged 1 commit into
masterfrom
fix/never-crash-resilience
May 30, 2026
Merged

Make solismon resilient: never crash, retry until recovered#1
dnviti merged 1 commit into
masterfrom
fix/never-crash-resilience

Conversation

@dnviti

@dnviti dnviti commented May 30, 2026

Copy link
Copy Markdown
Owner

Summary

Makes the monitor never crash. Any error is now logged and the process keeps running, actively retrying until it recovers and then continuing its work. Previously, routine transient conditions permanently killed the daemon.

Why

  • scrape_solis() called exit(1) on a connect failure or after 3 failed reads. Because from sys import exit raises SystemExit (a BaseException), it bypassed every except Exception and killed the process — so the inverter being briefly unreachable (overnight, Wi-Fi blip, reboot) permanently stopped monitoring.
  • The top-level handler also exit(1)'d, which under Home Assistant boot: auto becomes a restart/crash-loop.
  • A fresh Modbus socket (+ background reader thread) was opened every cycle and never closed → fd/thread leak that eventually crashes a long-running host.
  • SIGTERM (what Docker/HA send to stop) killed the process at the C level with no clean-shutdown path.

Changes

main.py

  • Remove all exit()/SystemExit paths; scrape_solis() logs and returns True/False. Bounded read retries return False (no exit, no hot-loop, no stale/unbound regs).
  • Run scraping in a background daemon thread (scrape_loop) that retries with exponential backoff capped at CHECK_INTERVAL on failure and resets to normal cadence on success — recovers fast, never hangs or hammers, and can never die.
  • Guarantee Modbus teardown in a finally (fixes the leak) and always disconnect the MQTT client.
  • Build metrics into a local dict and publish atomically on success, so a failed cycle keeps the last-good snapshot (and avoids a dict-resize race).
  • Prometheus collect() now only serves the latest snapshot (instant, never blocks/raises); exporter startup retries a port-bind failure instead of exiting.
  • Guard timestamp parsing and register indexing against malformed frames.
  • Install SIGTERM/SIGINT handlers for clean, logged shutdown.

config/config.py

  • Config parsing is total: a bad env var / option logs a warning and falls back to its default instead of raising at import time.
  • Clamp CHECK_INTERVAL to >= 1 (prevents sleep(-30) crashing and 0 busy-looping).

Behavior change

Prometheus mode now polls the inverter every CHECK_INTERVAL in a background thread (and /metrics serves the latest snapshot instantly), instead of polling only when scraped. This is what enables retry-until-recovered without hanging HTTP requests.

Verification

  • Unit tests: connect failure, persistent read failure, short/extra register frames, MQTT broker down, missing modified-metric inputs, and the full backoff loop (grow → cap → reset → never-raise → clean stop) — all pass.
  • Live runs of main.py with a mock inverter that fails twice then recovers: it retried, recovered, kept publishing, and on SIGTERM exited 0 with no traceback — in both MQTT and Prometheus modes.

🤖 Generated with Claude Code

The monitor could be permanently killed by routine, transient conditions
(inverter briefly unreachable, flaky Wi-Fi stick, MQTT broker down) and
would slowly leak sockets/threads on a long-running host. This reworks the
app so any error is logged and the process keeps running, actively retrying
until it recovers and then continuing its work.

main.py:
- Remove all exit()/SystemExit paths; scrape_solis() now logs and returns
  True/False instead of terminating. Bounded read retries return False
  (no exit, no hot-loop, no stale/unbound regs consumption).
- Run scraping in a background daemon thread (scrape_loop) that retries with
  exponential backoff capped at CHECK_INTERVAL on failure and resets to the
  normal cadence on success, so it recovers fast and never hangs or hammers.
- Guarantee Modbus connection teardown in a finally (fixes fd/thread leak)
  and always disconnect the MQTT client.
- Build metrics into a local dict and publish atomically on success, so a
  failed cycle keeps the last good snapshot (and avoids a dict-resize race).
- Prometheus collect() now only serves the latest snapshot (instant, never
  blocks/raises); the exporter start retries a port-bind failure instead of
  exiting.
- Guard timestamp parsing and register indexing against malformed frames.
- Install SIGTERM/SIGINT handlers for clean, logged shutdown (Docker/Home
  Assistant stop with SIGTERM, which previously killed it with no cleanup).

config/config.py:
- Make config parsing total: a bad env var / option logs a warning and falls
  back to the default instead of raising at import time.
- Clamp CHECK_INTERVAL to >= 1 to prevent a negative value crashing sleep()
  and a zero value busy-looping.

Co-Authored-By: Claude Opus 4.8 (1M context) <noreply@anthropic.com>
Copilot AI review requested due to automatic review settings May 30, 2026 17:21
@dnviti
dnviti merged commit 3e6ecdd into master May 30, 2026
1 check failed
@dnviti
dnviti deleted the fix/never-crash-resilience branch May 30, 2026 17:23

Copilot AI left a comment

Copy link
Copy Markdown

Choose a reason for hiding this comment

The reason will be displayed to describe this comment to others. Learn more.

Copilot encountered an error and was unable to review this pull request. You can try again by re-requesting a review.

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Labels

None yet

Projects

None yet

Development

Successfully merging this pull request may close these issues.

2 participants