loki.write WAL increases the marker file on restart even if the logs were not succesfully sent

[NHristovski]

Component(s)

loki.write

What's wrong?

We are testing the Alloy - loki.write WAL feature to make sure we never lose logs if Loki is unavailable ( returns 5xx or 429 HTTP status codes ).

We have two loki.write components that are sending logs on different urls and both components have WAL configured.

loki.write 1 sends the logs to endpoint that always returns HTTP 200 , and loki.write 2 sends the logs to endpoint that always returns HTTP 429.

When we restart the alloy instance ( or when the reload endpoint is called ) alloys goes into draining mode and tries to send all the logs from the WAL.
Component loki.write 2 can not send the logs ( because it always gets the HTTP 429 ) but the marker for the WAL is incremented to the latest file, so when the alloy restarts the logs that could not be sent are lost because the WAL marker is at the latest position.

We would like to request the WAL marker to NOT be advanced if alloy gets retry-able HTTP code when doing the re-sending ( 429 / 5xx )

Steps to reproduce

1. Configure alloy to send logs for pod1 to endpoint1 ( this endpoint should return http 200 all the time ) and the logs for pod2 to endpoint2 ( this enpoint should return http 429 all the time ) and enable the WAL for both loki.write components ( see the configuration section )

2. Open a shell to the alloy pod and make sure that the WAL marker is on the same position for the endpoint2 writer.

3. Restart the pod and then open a shell again and check the WAL marker for the endpoint2 writer -> it will be on the latest position but the logs will be missing in Loki

System information

AWS EKS, AWS Linux - 64-bit Arm

Software version

Grafana Alloy v1.19.2

Configuration

loki.write "loki_write_success" {
      endpoint {
        
        url = "<endpoint_that_returns_200>"
        
        bearer_token_file = "<path-to-bearer-token>"


        batch_wait          = "15s"
        batch_size          = "4MiB"
        retry_on_http_429   = true
        min_backoff_period  = "10s"
        max_backoff_period  = "10m0s"
        max_backoff_retries = 10000000
      }
      external_labels = {}

      wal {
        enabled            = true
        max_segment_age    = "2h"
        drain_timeout      = "30s"
        min_read_frequency = "250ms"
        max_read_frequency = "1s"
      }
    }

    loki.write "loki_write_failed" {
      endpoint {
        
        url = "<endpoint_that_returns_429>"
        
        bearer_token_file = "<path-to-bearer-token>"

        batch_wait          = "15s"
        batch_size          = "4MiB"
        retry_on_http_429   = true
        min_backoff_period  = "10s"
        max_backoff_period  = "10m0s"
        max_backoff_retries = 10000000
      }
      external_labels = {}

      wal {
        enabled            = true
        max_segment_age    = "2h"
        drain_timeout      = "30s"
        min_read_frequency = "250ms"
        max_read_frequency = "1s"
      }
    }

Logs

AI disclosure

• This issue was substantially generated with AI assistance.

Tip

_{React with 👍 if this issue is important to you.}

[kalleep]

Hey, yes as of today this is intended i.e. we always mark data as sent after all retries have been made. So I would not mark this is a bug but rather a enhancement / feature request.

There are scenarios where we should prevent marker from advancing, like 429, but there is also scenarios we we need to increase advance the marker e.g. status 400 etc since this data can never be accepted by loki and we need to move past it.

[NHristovski]

Thank you for the reply @kalleep .

Should I close this issue, and open a new one as a request so that the marker is never advanced on 429 or 5xx?

[kalleep]

No it's fine to keep this open, if you want you can change title and description to better match the feature request.

Another thing to consider is if this should be configurable or not..

[NHristovski]

The description was changed to be clear that this is a request for new feature / enhancement and not a bug report.

For the configuration part, I would leave the community to decide.

From my point of view: We try to never lose logs, and we rely on the retry logic a lot, so for us it makes sense that by default the marker should NOT be advanced if Loki is sending retryable codes ( 429/5xx ). But this will mean that the disk can get full if Loki is down for a long period of time, so it makes sense to also provide configuration parameter to disable this feature, and the marker to be advanced no matter if the logs were successfully sent or not

[kalleep]

Thought a bit more about this and I think it should be classified as a bug. We already retry on 429 and 5XX and we do always advance the marker after all retries have been exhausted and that is all fine.

But during shutdown we try to drain the queue with a timeout and after that timeout is finished we cancel everything and marker gets advances. So the fix would be to not advance the marker during shutdown and then you as a user can always correctly control how long you want to retry with the options we already have.