Flapping Alerts

[marcusboden]

Some of the alerts from this charm tend to be flapping a lot. The CollectorFailed alert is prone this, because it's very dependent on the hardware of the BMC, which tend to be flaky. This results in really annoying, flapping alerts, if the BMC doesn't respond in time.

Can we improve the alertrule by adding the keep_firing_for option and have it always at least fire for an hour?
From the docs:

There is also an optional keep_firing_for clause that tells Prometheus to keep this alert firing for the specified duration after the firing condition was last met. This can be used to prevent situations such as flapping alerts, false resolutions due to lack of data loss, etc. Alerting rules without the keep_firing_for clause will deactivate on the first evaluation where the condition is not met (assuming any optional for duration described above has been satisfied).

[syncronize-issues-to-jira]

Thank you for reporting your feedback to us!

The internal ticket has been created: https://warthogs.atlassian.net/browse/SOLENG-1641.

This message was autogenerated

[gabrielcocenza]

From my understanding the keep_firing_for won't reduce the noise because it will keep alerting event that the collector back to normal.

Maybe we should increase the time to start triggering if the collector is not working for something like 15 or 30 minutes.

@marcusboden can you confirm that you want to have an alert triggering for like an hour even that the collector starts collecting metrics again ?

Do you know another alerts that are bothering operations?

[marcusboden]

From my understanding the keep_firing_for won't reduce the noise because it will keep alerting event that the collector back to normal.

Yes, that's exactly what I want. The issue is that some of the collectors response time hover around the timeout time, resulting in e.g. 20% of the metric missing, which in turn triggers some alerts sometimes (if it happens longer than 5min by chance), but most of the time not. But it's super annoying and hard to silence, as the moment you look in alertmanager, they are resolved already and you have to build the correct set of tags for the silence from memory.

However, I also agree that we should increase the for: part to 30 min. We usually don't really care if the IPMI doesn't return perfectly on time every few minutes.

Small addendum: I already tried to define my own alert for that here, but there may be a bug with the keep_firing_for in the cos library. Just as a heads-up, you may run into this as well.

[gabrielcocenza]

#481 is targeting the for part and to use keep_firing_for with need the observability team to update the Prometheus module that they are using for cos-tool

[Pjack]

We couldn't add keep_firing_for option now because of canonical/prometheus-k8s-operator#746. The alert rule will become invalid and been dropped by prometheus. We have to wait for the issue been fixed.

[gabrielcocenza]

Just an update. cos-tool has updated the Prometheus module, however they are still not available at prometheus 1/stable or 2/stable