mixin: Create Loki Pipeline Alerts

[kgeckhart]

Component(s)

No response

Request

As a part of, #7058 we introduced some special handling so it was obvious when a Loki batch was too large. I asked about adding an alert so we could be proactive and we discovered we have no loki pipeline alerts which seems like a big gap.

Alerts we know we want

I want to know when my Loki pipelines are misbehaving proactively. Use cases for pipeline alerts,

Symptoms - critical

• Success rate of writing to Loki going low - logs will be delayed and could end up missing
• Propagation delay growing high - logs will be delayed and could end up missing
• Log source falling behind - logs will be delayed and could end up missing

Causes - warning

• Request batch sizes are too big (feat: Add special reason when Loki returns 413 #7058) leading to data loss

Tip

_{React with 👍 if this issue is important to you.}

[thampiotr]

Copying here what I wrote on Slack earlier today:

I think that we want to start with something alerting on symptoms only, not the causes:

Success rate of writing to Loki going low
Propagation delay growing high
That should cover most issues. I don't think we need a batch size too large alert. It's an alert on a cause. I think that you would get alerted on success rate low and then you can see in logs the cause and fix it.

[kgeckhart]

As discussed in slack I reformatted the issue around symptoms and causes where symptoms are critical and causes are warning.

The goal of causes in these cases are for known things that should be covered by a symptom but can either be an early warning signal or be a very quick root cause analysis shortcut in the event the a symptom alert fires.