Avoid error logs flooding when connecting to a database is failed

[envzhu]

Problem

When connecting to a database is failed in write_tsdb plugin , error logs floods on the collectd client like below:

Modification

To avoid this, I changed logging code in write_tsdb.c based on write_graphite.c.

• Use c_complain() and c_release() instead of ERROR() to output the connecting error log
  • By these function, the error interval increases by two times every time the connecting error is logged . (Actually, the interval is limited to 86400 second.)
• Delete error logs just after wt_callback_init() and wt_send_message() since an error message has already been printed by this log.

Test

Configure collectd.conf to connect to unexist database intendedly.

Interval 10

LoadPlugin syslog
<Plugin "syslog">
    LogLevel err
</Plugin>

LoadPlugin cpu

LoadPlugin write_tsdb
<Plugin "write_tsdb">
    <Node>
        Host "localhost"
        Port "4242"
        DeriveRate true
    </Node>
</Plugin>

Before this patch

# systemctl status collectd -l

Aug 31 08:31:59 collectd-centos8 collectd[2867]: write_tsdb plugin: wt_callback_init failed.
Aug 31 08:31:59 collectd-centos8 collectd[2867]: write_tsdb plugin: error with wt_send_message
Aug 31 08:31:59 collectd-centos8 collectd[2867]: write_tsdb plugin: wt_callback_init failed.
Aug 31 08:31:59 collectd-centos8 collectd[2867]: write_tsdb plugin: error with wt_send_message
Aug 31 08:31:59 collectd-centos8 collectd[2867]: write_tsdb plugin: wt_callback_init failed.
Aug 31 08:31:59 collectd-centos8 collectd[2867]: write_tsdb plugin: error with wt_send_message
Aug 31 08:31:59 collectd-centos8 collectd[2867]: write_tsdb plugin: wt_callback_init failed.
Aug 31 08:31:59 collectd-centos8 collectd[2867]: write_tsdb plugin: error with wt_send_message
Aug 31 08:31:59 collectd-centos8 collectd[2867]: write_tsdb plugin: wt_callback_init failed.
Aug 31 08:31:59 collectd-centos8 collectd[2867]: write_tsdb plugin: error with wt_send_message

Error logs flooded.

After this patch

# systemctl status collectd -l

Aug 31 07:44:57 collectd-centos8 systemd[1]: Starting Collectd statistics daemon...
Aug 31 07:44:57 collectd-centos8 collectd[1887]: plugin_load: plugin "syslog" successfully loaded.
Aug 31 07:44:57 collectd-centos8 systemd[1]: Started Collectd statistics daemon.
Aug 31 07:45:17 collectd-centos8 collectd[1887]: write_tsdb plugin: Connecting to localhost:4242 failed. The last error was: Connection refused
Aug 31 07:45:27 collectd-centos8 collectd[1887]: write_tsdb plugin: Connecting to localhost:4242 failed. The last error was: Connection refused
Aug 31 07:45:47 collectd-centos8 collectd[1887]: write_tsdb plugin: Connecting to localhost:4242 failed. The last error was: Connection refused
Aug 31 07:46:27 collectd-centos8 collectd[1887]: write_tsdb plugin: Connecting to localhost:4242 failed. The last error was: Connection refused
Aug 31 07:48:37 collectd-centos8 collectd[1887]: write_tsdb plugin: Connecting to localhost:4242 failed. The last error was: Connection refused
Aug 31 07:51:17 collectd-centos8 collectd[1887]: write_tsdb plugin: Connecting to localhost:4242 failed. The last error was: Connection refused
Aug 31 07:56:57 collectd-centos8 collectd[1887]: write_tsdb plugin: Connecting to localhost:4242 failed. The last error was: Connection refused

You can see the error interval increases by two times every time the connecting error is logged