feat: enable out-of-order samples

[sinapah]

Issue

Fixes #185 and fixes #465.
Read this blog by upstream to understand what the design of this feature looks like: https://grafana.com/blog/new-in-grafana-mimir-introducing-out-of-order-sample-ingestion/. There is also a brief discussion on the tradeoffs of enabling it when it comes to CPU and memory use.

Solution

Add a charm config option called out_of_order_time_window, named after the upstream config name that is disabled by default. It's a string that accepts the time units s, m, etc.

Context

Testing Instructions

Deploy this bundle:

bundle: kubernetes
applications:
  aval:
    charm: avalanche-k8s
    channel: dev/edge
    revision: 80
    base: ubuntu@26.04/stable
    resources:
      avalanche-image: 21
    scale: 1
    constraints: arch=amd64
    trust: true
  grafana:
    charm: grafana-k8s
    channel: dev/edge
    revision: 199
    base: ubuntu@26.04/stable
    resources:
      grafana-image: 79
    scale: 1
    constraints: arch=amd64
    storage:
      database: kubernetes,1,1024M
    trust: true
  mimir:
    charm: mimir-worker-k8s
    channel: dev/edge
    revision: 119
    resources:
      nginx-image: 16
      nginx-prometheus-exporter-image: 5
    scale: 3
    constraints: arch=amd64
    trust: true
  mimir-backend:
    charm: mimir-worker-k8s
    channel: dev/edge
    revision: 108
    base: ubuntu@26.04/stable
    resources:
      mimir-image: 18
    scale: 3
    options:
      role-backend: true
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
      recovery-data: kubernetes,1,1024M
    trust: true
  mimir-read:
    charm: mimir-worker-k8s
    channel: dev/edge
    revision: 108
    base: ubuntu@26.04/stable
    resources:
      mimir-image: 18
    scale: 3
    options:
      role-read: true
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
      recovery-data: kubernetes,1,1024M
    trust: true
  mimir-write:
    charm: mimir-worker-k8s
    channel: dev/edge
    revision: 108
    base: ubuntu@26.04/stable
    resources:
      mimir-image: 18
    scale: 3
    options:
      role-write: true
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
      recovery-data: kubernetes,1,1024M
    trust: true
  otelcol:
    charm: opentelemetry-collector-k8s
    channel: dev/edge
    revision: 253
    base: ubuntu@26.04/stable
    resources:
      opentelemetry-collector-image: 11
    scale: 3
    constraints: arch=amd64
    storage:
      persisted: kubernetes,1,1024M
    trust: true
  swfs:
    charm: seaweedfs-k8s
    channel: latest/edge
    revision: 9
    resources:
      seaweedfs-image: 2
    scale: 1
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
    trust: true
  trfk:
    charm: traefik-k8s
    channel: latest/edge
    revision: 449
    base: ubuntu@20.04/stable
    resources:
      traefik-image: 177
    scale: 1
    constraints: arch=amd64
    storage:
      configurations: kubernetes,1,1024M
    trust: true
relations:
- - mimir:s3
  - swfs:s3-credentials
- - mimir:mimir-cluster
  - mimir-write:mimir-cluster
- - mimir:mimir-cluster
  - mimir-read:mimir-cluster
- - mimir:mimir-cluster
  - mimir-backend:mimir-cluster
- - aval:metrics-endpoint
  - otelcol:metrics-endpoint
- - otelcol:send-remote-write
  - mimir:receive-remote-write
- - mimir:grafana-source
  - grafana:grafana-source
- - grafana:ingress
  - trfk:ingress

1. Simulate the issue by adding a lag batch processor the pipeline scraping Aval metrics and remote writing them into Mimir. For this purpose, go to only one of the three Otelcol units and replace the workload config with something like: https://pastebin.canonical.com/p/ksx3cDm6J8/. The effect of this is that while the other two Otelcol units continue to scrape and remote write with no lag, Otelcol/0 (or any other unit) lags behind 2 minutes. As a result, by the time it writes sample x, the sample has already been ingested roughly 2 minutes prior.
2. After a couple of minutes, you should start to see rejections in Mimir logs that look like:

2026-09-11T15:21:34.146Z [mimir] ts=2026-09-11T15:21:34.0704484Z caller=push.go:251 level=error user=anonymous msg="detected an error while ingesting Prometheus remote-write request (the request may have been partially ingested)" httpCode=400 err="send data to ingesters: failed pushing to ingester mimir-write-1: user=anonymous: the sample has been rejected because another sample with a more recent timestamp has already been ingested and out-of-order samples are not allowed (err-mimir-sample-out-of-order). The affected sample has timestamp 2026-09-11T15:20:33.878Z and is from series avalanche_gauge_metric_mmmmm_0_329{cycle_id=\"0\", instance=\"test_acbc520f-ad5b-443d-8ffe-61f02c5c564f_aval_aval/0\", job=\"juju_test_acbc520f_aval_prometheus_scrape_aval-0\", juju_application=\"aval\", juju_charm=\"avalanche-k8s\", juju_model=\"test\", juju_model_uuid=\"acbc520f-ad5b-443d-8ffe-61f02c5c564f\", juju_unit=\"aval/0\", label_key_kkkkk_0=\"label_val_vvvvv_0\", label_key_kkkkk_1=\"label_val_vvvvv_1\", label_key_kkkkk_2=\"label_val_vvvvv_2\", label_key_kkkkk_3=\"label_val_vvvvv_3\", label_key_kkkkk_4=\"label_val_vvvvv_4\", label_key_kkkkk_5=\"label_val_vvvvv_5\", label_key_kkkkk_6=\"label_val_vvvvv_6\", label_key_kkkkk_7=\"label_val_vvvvv_7\", label_key_kkkkk_8=\"label_val_vvvvv_8\", label_key_kkkkk_9=\"label_val_vvvvv_9\", series_id=\"8\"} (sampled 1/10)" insight=true

3. Now, refresh the coordinator charm with changes in this PR.
4. juju config mimir out_of_order_time_window=5m.
5. The errors should go away.

This is what the Grafana Explore page should look like.

Performance concerns

To test whether enabling this time window negatively and noticeably impacts memory and CPU use among Mimir worker pods, I used sudo k8s kubectl top to continously monitor CPU and memory use at 10s intervals for 15 minutes before (baseline) and after the change. The CSV files are attached here.
mimir_resource_usage_baseline.csv
mimir_resource_usage_after.csv
I then plotted the CPU and memory use comparison plots.
Similar to the findings in https://grafana.com/blog/new-in-grafana-mimir-introducing-out-of-order-sample-ingestion/, memory use didn't noticeably change. In both the read and write pods before and after the change, memory fluctuated in ways that implied the enabling of out-of-order sample ingestion was not to blame.

A minor note here is that as this experiment was going on, I was querying the Mimir datasource in Grafana automatically ever 5s. This querying flow would test the memory use of the Mimir read units.
The same is more or less the same for CPU.

Upgrade Notes

This needs to be backported to track 3 and also implemented in Prometheus (canonical/prometheus-k8s-operator#564).

[MichaelThamm]

1. out_of_order_time_window is passed through as a raw string, unlike retention_period which is gated by is_valid_timespec + BlockedStatus. juju config mimir out_of_order_time_window=5min is invalid Mimir YAML causing a worker crashloop. Reuse cosl.time_validation.is_valid_timespec.
2. Doc string is copy-pasted from retention and wrong: "Supported units: d, w, m, y…". in Prometheus durations m is minutes; y/w are nonsense for an OOO window. Also: upstream still marks -ingester.out-of-order-time-window as experimental in Mimir 2.17.
3. 5m would not have fixed [Urgent] Metrics pipeline stalled on Central COS; root cause under investigation #185. The rejected samples there were 1h45m old (timestamp-too-old, not just OOO). Guidance must be "window ≥ worst-case collector backlog age", which with otelcol's persistent queues is hours, not minutes - at least a doc item would help.
4. Missing the actually valuable bit: an alert. There is no rule on cortex_discarded_samples_total{reason="sample-out-of-order"|"sample-timestamp-too-old"}. Silent data loss is the real failure mode in both [Urgent] Metrics pipeline stalled on Central COS; root cause under investigation #185 and Out of order sample from remote write prometheus-k8s-operator#564. Add it in this PR. We do have the failed-logs alert in Otelcol which is applicable, but not specific to these errors. Is it worth adding to Mimir (and all datastores) vs. relying on Otelcol?
5. The != "0s" string guard is brittle ("0", "0m" bypass it) and unnecessary. Mimir treats 0 as disabled, so just always emit the value.

[MichaelThamm]

Some things to consider:

• https://github.com/grafana/mimir/blob/88dbc5d983683cdfbb655355c5fad69bdc85304c/cmd/mimir/help-all.txt.tmpl#L1694

  • reduces pressure on compactors and ingesters when the out-of-order samples span multiple days in the past.

• https://grafana.com/docs/mimir/latest/configure/configure-out-of-order-samples-ingestion/#recording-rules-when-out-of-order-ingestion-is-enabled
  • The link in your docs PR covers this, just FYI for us to know
• -validation.enforce-out-of-order-window-on-distributor (default false and experimental) might be worth exposing/defaulting on: rejects at the distributor instead of partially ingesting and failing at N ingesters. Cheaper, clearer errors.
• Otelcol (the sender side: where the problem should actually be solved) currently only queue_size is exposed. Consider exposing/pinning remote_write_queue.num_consumers: 1 (restores per-series ordering for PRW) and retry_on_failure.max_elapsed_time (hard-coded 5m), and document that scaling an otelcol app duplicates every scrape job rather than sharding it.