docs: Add topic about Loki pipeline resiliency

[clayton-cornell]

Component(s)

No response

Add a description

loki.write WaL is tagged experimental which some customers view as a blocker for production deployment plus there isn't a lot of info about how this actually works and how it can help with network resiliency (outage survival).

No existing doc clearly explains:

• What actually happens to log delivery during an outage without WAL
• What WAL changes about that behavior
• Where the practical limits are (Loki's ingestion time window) even with WAL
• Which behaviors are tied to the experimental queue_config vs. stable defaults

A new topic:

Loki pipeline resiliency during endpoint outages

1. Why it matters (network hiccups, maintenance, endpoint outages... other?)
2. Default behaviour without WAL.
3. How queue_config works.
4. What WAL changes/fixes/helps with.
5. Loki ingestion limits that affect buffering
6. Best practices
7. Links to/from related topics/component reference etc.

Tip

_{React with 👍 if this issue is important to you.}

[clayton-cornell]

Additional notes from Slack discussion

• Limits - How old an entry will Loki ingest if the outage is protracted?
• Resource Consumption - What resources (disk, memory, etc) will be consumed by WAL during an outage?  Can we limit their consumption?  If consumption is limited how does WAL behave (fill until full and then drop, ring buffer, etc)
• Avoiding Data Loss - Once connectivity is re-established, how do we avoid any rate limiting behaviors that would result in data loss.

(In this context, outage = "Grafana cloud becomes unreachable"