From 7f7d39db26227bf5e0a9cdcfd61260988d43da8a Mon Sep 17 00:00:00 2001
From: Sina P <55766091+sinapah@users.noreply.github.com>
Date: Fri, 11 Sep 2026 16:48:00 -0400
Subject: [PATCH 1/3] feat(docs): troubleshooting out-of-order samples in Mimir
 (#486)

* feat(docs): troubleshooting out-of-order samples in Mimir

* fix: address comments
---
 .../troubleshooting.md                        | 39 +++++++++++++++++++
 1 file changed, 39 insertions(+)

diff --git a/docs/how-to/validate-and-troubleshoot/troubleshooting.md b/docs/how-to/validate-and-troubleshoot/troubleshooting.md
index 7e499923..aa8355dc 100644
--- a/docs/how-to/validate-and-troubleshoot/troubleshooting.md
+++ b/docs/how-to/validate-and-troubleshoot/troubleshooting.md
@@ -572,6 +572,45 @@ pipeline, or Tempo on the traces pipeline) are *not* dropped and still reach Lok
 form a loop while the logs path is healthy, so their `Exporting failed` logs remain visible in
 Grafana.
 
+## `err-mimir-sample-out-of-order` and `err-mimir-sample-timestamp-too-old`
+
+For any given series, Mimir requires each sample's timestamp to be newer than the last
+one already ingested. Samples that violate this are rejected by default:
+
+```
+the sample has been rejected because another sample with a more recent timestamp
+has already been ingested and out-of-order samples are not allowed (err-mimir-sample-out-of-order)
+```
+
+Timestamps are assigned when telemetry is *collected*, but ordering is only checked when it
+is *written*. Anything that decouples the two produces this error, so identify the cause
+before widening the accepted window.
+
+### Common causes
+1. **More than one collector writing the same series**. Scaling up `opentelemetry-collector` does not shard scrape jobs: every unit scrapes every target, so all units emit identical series. Whenever one unit falls behind its peers its samples are older than what the others have already written.
+2. **A backlog flushed after an outage.** Collector queues hold telemetry with its original timestamps. When connectivity is restored, live data usually lands first and the backlog arrives behind it.
+3. **Concurrent writes from a single collector.** The OpenTelemetry Collector's `prometheusremotewrite` exporter sends multiple requests in parallel and provides no per-series ordering guarantee.
+4. **Clock skew between collector hosts.** Samples are timestamped with the collecting host's clock, so a host whose clock lags its peers writes samples that appear old.
+5. **Uneven pipeline delay.** A `batch` processor with a long `timeout`, or `memory_limiter` backpressure, applied to some collectors but not others creates the same asymmetry as cause 1.
+
+### Resolve
+
+If out-of-order writes are unavoidable, allow an out-of-order window on the Mimir coordinator:
+
+```bash
+juju config mimir out_of_order_time_window=5m
+```
+
+```{note}
+The same setting raises the bound for `err-mimir-sample-timestamp-too-old`; there is no separate option for that error. Setting the value back to `0s` disables out-of-order ingestion again, while out-of-order samples already ingested remain queryable.
+```
+
+```{note}
+Enabling out-of-order sample ingestion may lead to a CPU usage increase and a minor memory increase. See the [upstream documentation](https://grafana.com/docs/mimir/latest/configure/configure-out-of-order-samples-ingestion/#configure-out-of-order-samples-ingestion) for more details on tradeoffs.
+```
+
+The value is a duration string (e.g. `5m`, `10m`, `1h`). Choose a window that covers the expected delay between sample generation and ingestion.
+
 ## `socket: too many open files`
 
 When deploying the Opentelemetry Collector or Prometheus charms in large environments,

From 46406c6290ba6cd9a5d837dbc40dab2f62c4da91 Mon Sep 17 00:00:00 2001
From: Sina <sina.pahlavan@canonical.com>
Date: Fri, 11 Sep 2026 16:54:03 -0400
Subject: [PATCH 2/3] fix: spelling

---
 docs/reference/security-hardening-guide.md | 2 +-
 1 file changed, 1 insertion(+), 1 deletion(-)

diff --git a/docs/reference/security-hardening-guide.md b/docs/reference/security-hardening-guide.md
index a4b7574b..10beb7ac 100644
--- a/docs/reference/security-hardening-guide.md
+++ b/docs/reference/security-hardening-guide.md
@@ -53,6 +53,6 @@ For cases where:
 
 It may be possible to secure the entire ingress with authentication.  For example, see the [basic authentication](https://charmhub.io/traefik-k8s/configurations#basic_auth_user) and [`forward_auth`](https://charmhub.io/traefik-k8s/configurations#enable_experimental_forward_auth) integrations on the Traefik charm.
 
-## Secure configuation
+## Secure configuration
 Use Juju secrets where applicable. For example, the `opentelemetry-collector-integrator` can be used for forwarding exporter
 configuration to `opentelemetry-collector`. Do not pass secrets, such as token in cleartext; use Juju secrets instead.

From cd5a2d725be88a6be254957cd556568fd5d11ab8 Mon Sep 17 00:00:00 2001
From: Sina <sina.pahlavan@canonical.com>
Date: Mon, 14 Sep 2026 08:55:47 -0400
Subject: [PATCH 3/3] fix: Unrelate spelling

---
 docs/.custom_wordlist.txt | 1 +
 1 file changed, 1 insertion(+)

diff --git a/docs/.custom_wordlist.txt b/docs/.custom_wordlist.txt
index eff08a07..04933027 100644
--- a/docs/.custom_wordlist.txt
+++ b/docs/.custom_wordlist.txt
@@ -217,6 +217,7 @@ UIs
 uncharmed
 uncomment(ing)?
 unencrypted
+Unrelate
 URL
 utils
 UUID