Skip to content

ingestor: watchdog force-reconnect logs a misleading Connect() error while paho is in its initial ConnectRetry loop #102

Description

@dborup

Problem

After #29 (port of upstream Kpa-clawbot/CoreScope#1897), the MQTT watchdog's force-reconnect logs

MQTT [<tag>] WATCHDOG force-reconnect Connect() failed: status can only transition to connecting from disconnected

on every trigger while paho is in its initial ConnectRetry loop (status connecting).

Why this is misleading

  • It is functionally harmless: paho's own retry loop keeps running, which is the intended behaviour. The port(upstream#1897): stop watchdog force-reconnect racing paho's retry loop #29 review measured 5/5 triggers in connecting, each logging the error, with 0 extra CONNECT packets.
  • To the watchdog, connecting looks the same as reconnecting (IsConnected=true, IsConnectionOpen=false). In reconnecting, Connect() is a genuine no-op.
  • The line is rate-limited by the force-reconnect throttle to about 1 per minute per source. But it appears exactly during the outages when operators read the logs, and it reads like a real failure.
  • The code comment in cmd/ingestor/main.go (around line 600) says Connect() alone is "a safe no-op per paho when a retry is already under way". That is true for reconnecting but not for connecting.

Suggested fix

  • Recognise the "status can only transition to connecting from disconnected" error (or check paho's connection status before calling Connect()), and log it at debug level or as an informational "retry already in progress" line. Genuine Connect() failures should still be logged as errors.
  • Correct the code comment.
  • Add a test with a real paho client in connecting state. The in-test broker from cmd/ingestor/mqtt_force_reconnect_paho_test.go can be reused. The test should assert that no error-level line is emitted, while a genuine failure still is.

Found in the independent review of #29 (SHOULD-FIX, non-blocking).

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions