Skip to content

feat (track 2.17): enable out-of-order samples - #469

Merged
sinapah merged 1 commit into
track/2.17from
feat/track-2.17/enable-out-of-order-samples
Sep 14, 2026
Merged

feat (track 2.17): enable out-of-order samples#469
sinapah merged 1 commit into
track/2.17from
feat/track-2.17/enable-out-of-order-samples

Conversation

@sinapah

@sinapah sinapah commented Sep 14, 2026 •
edited
Loading

Copy link
Copy Markdown
Contributor

Issue

Backport of #468.

Solution

Add a charm config option that allows setting the "out_of_order_window".

Context

Testing Instructions

Before

To observe the "before" behaviour, deploy:

bundle: kubernetes
applications:
  aval:
    charm: avalanche-k8s
    channel: dev/edge
    revision: 80
    base: ubuntu@26.04/stable
    resources:
      avalanche-image: 21
    scale: 1
    constraints: arch=amd64
    trust: true
  grafana:
    charm: grafana-k8s
    channel: dev/edge
    revision: 199
    base: ubuntu@26.04/stable
    resources:
      grafana-image: 79
    scale: 1
    constraints: arch=amd64
    storage:
      database: kubernetes,1,1024M
    trust: true
  mimir:
    charm: mimir-coordinator-k8s
    channel: 2.17/edge
    revision: 118
    base: ubuntu@26.04/stable
    resources:
      nginx-image: 16
      nginx-prometheus-exporter-image: 5
    scale: 3
    constraints: arch=amd64
    trust: true
  mimir-backend:
    charm: mimir-worker-k8s
    channel: 2.17/edge
    revision: 103
    base: ubuntu@26.04/stable
    resources:
      mimir-image: 18
    scale: 3
    options:
      role-backend: true
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
      recovery-data: kubernetes,1,1024M
    trust: true
  mimir-read:
    charm: mimir-worker-k8s
    channel: 2.17/edge
    revision: 103
    base: ubuntu@26.04/stable
    resources:
      mimir-image: 18
    scale: 3
    options:
      role-read: true
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
      recovery-data: kubernetes,1,1024M
    trust: true
  mimir-write:
    charm: mimir-worker-k8s
    channel: 2.17/edge
    revision: 103
    base: ubuntu@26.04/stable
    resources:
      mimir-image: 18
    scale: 3
    options:
      role-write: true
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
      recovery-data: kubernetes,1,1024M
    trust: true
  otelcol:
    charm: opentelemetry-collector-k8s
    channel: dev/edge
    revision: 253
    base: ubuntu@26.04/stable
    resources:
      opentelemetry-collector-image: 11
    scale: 3
    constraints: arch=amd64
    storage:
      persisted: kubernetes,1,1024M
    trust: true
  swfs:
    charm: seaweedfs-k8s
    channel: latest/edge
    revision: 9
    resources:
      seaweedfs-image: 2
    scale: 1
    constraints: arch=amd64
    storage:
      data: kubernetes,1,1024M
    trust: true
  trfk:
    charm: traefik-k8s
    channel: latest/edge
    revision: 449
    base: ubuntu@20.04/stable
    resources:
      traefik-image: 177
    scale: 1
    constraints: arch=amd64
    storage:
      configurations: kubernetes,1,1024M
    trust: true
relations:
- - mimir:s3
  - swfs:s3-credentials
- - mimir:mimir-cluster
  - mimir-write:mimir-cluster
- - mimir:mimir-cluster
  - mimir-read:mimir-cluster
- - mimir:mimir-cluster
  - mimir-backend:mimir-cluster
- - aval:metrics-endpoint
  - otelcol:metrics-endpoint
- - otelcol:send-remote-write
  - mimir:receive-remote-write
- - mimir:grafana-source
  - grafana:grafana-source
- - grafana:ingress
  - trfk:ingress

Then, to quickly reproduce the issue, replace the workload config of ONE of the three Otelcols to:

connectors: {}
exporters:
  nop/otelcol/0: {}
  prometheusremotewrite/send-remote-write/0:
    add_metric_suffixes: false
    endpoint: http://mimir-0.mimir-endpoints.test.svc.cluster.local:8080/api/v1/push
    remote_write_queue:
      enabled: true
      queue_size: 1000
    retry_on_failure:
      max_elapsed_time: 5m
    tls:
      insecure_skip_verify: false
  prometheusremotewrite/send-remote-write/1:
    add_metric_suffixes: false
    endpoint: http://mimir-1.mimir-endpoints.test.svc.cluster.local:8080/api/v1/push
    remote_write_queue:
      enabled: true
      queue_size: 1000
    retry_on_failure:
      max_elapsed_time: 5m
    tls:
      insecure_skip_verify: false
  prometheusremotewrite/send-remote-write/2:
    add_metric_suffixes: false
    endpoint: http://mimir-2.mimir-endpoints.test.svc.cluster.local:8080/api/v1/push
    remote_write_queue:
      enabled: true
      queue_size: 1000
    retry_on_failure:
      max_elapsed_time: 5m
    tls:
      insecure_skip_verify: false
extensions:
  file_storage:
    directory: /otelcol
  health_check:
    endpoint: 0.0.0.0:13133
processors:
  attributes/send-loki-logs:
    actions:
    - action: upsert
      key: loki.attribute.labels
      value: container, job, filename, juju_application, juju_charm, juju_model, juju_model_uuid,
        juju_unit, snap_name, path, pebble_service
  batch/metrics-lag:
    timeout: 2m
    send_batch_size: 10000
  filter/internal-telemetry-loop-breaker/otelcol/0:
    error_mode: ignore
    logs:
      log_record: []
  resource/send-loki-logs:
    attributes:
    - action: insert
      key: loki.format
      value: raw
receivers:
  otlp/otelcol/0:
    protocols:
      grpc:
        endpoint: 0.0.0.0:4317
      http:
        endpoint: 0.0.0.0:4318
  prometheus/metrics-endpoint/otelcol/0:
    config:
      scrape_configs:
      - job_name: juju_test_acbc520f_aval_prometheus_scrape_aval-0
        metrics_path: /metrics
        relabel_configs:
        - regex: (.*)
          separator: _
          source_labels:
          - juju_model
          - juju_model_uuid
          - juju_application
          - juju_unit
          target_label: instance
        scrape_interval: 1m
        scrape_timeout: 10s
        static_configs:
        - labels:
            juju_application: aval
            juju_charm: avalanche-k8s
            juju_model: test
            juju_model_uuid: acbc520f-ad5b-443d-8ffe-61f02c5c564f
            juju_unit: aval/0
          targets:
          - aval-0.aval-endpoints.test.svc.cluster.local:9001
        tls_config:
          insecure_skip_verify: false
  prometheus/self-monitoring/otelcol/0:
    config:
      scrape_configs:
      - job_name: juju_test_acbc520f_otelcol_self-monitoring
        scrape_interval: 1m
        scrape_timeout: 10s
        static_configs:
        - labels:
            instance: test_acbc520f_otelcol_otelcol/0
            juju_application: otelcol
            juju_charm: opentelemetry-collector-k8s
            juju_model: test
            juju_model_uuid: acbc520f-ad5b-443d-8ffe-61f02c5c564f
            juju_unit: otelcol/0
          targets:
          - 0.0.0.0:8888
service:
  extensions:
  - health_check
  - file_storage
  pipelines:
    logs/otelcol/0:
      exporters:
      - nop/otelcol/0
      processors:
      - filter/internal-telemetry-loop-breaker/otelcol/0
      - resource/send-loki-logs
      - attributes/send-loki-logs
      receivers:
      - otlp/otelcol/0
    metrics/otelcol/0:
      exporters:
      - prometheusremotewrite/send-remote-write/0
      - prometheusremotewrite/send-remote-write/1
      - prometheusremotewrite/send-remote-write/2
      processors:
      - batch/metrics-lag
      receivers:
      - otlp/otelcol/0
      - prometheus/self-monitoring/otelcol/0
      - prometheus/metrics-endpoint/otelcol/0
    traces/otelcol/0:
      exporters:
      - nop/otelcol/0
      receivers:
      - otlp/otelcol/0
  telemetry:
    logs:
      disable_stacktrace: true
      level: INFO
      processors:
      - batch:
          exporter:
            otlp:
              endpoint: http://localhost:4318
              protocol: http/protobuf
    metrics:
      level: normal
    resource:
      juju_application: otelcol
      juju_charm: opentelemetry-collector-k8s
      juju_model: test
      juju_model_uuid: acbc520f-ad5b-443d-8ffe-61f02c5c564f
      juju_unit: otelcol/0
      loki.format: logfmt
      loki.resource.labels: juju_application, juju_charm, juju_model, juju_model_uuid,
        juju_unit
      service.instance.id: otelcol/0
      service.name: otelcol-internal

Then, the Mimir Write units should log errors such as:

2026-09-14T13:35:34.217Z [mimir] ts=2026-09-14T13:35:34.21546593Z caller=push.go:251 level=error user=anonymous msg="detected an error while ingesting Prometheus remote-write request (the request may have been partially ingested)" httpCode=400 err="send data to ingesters: failed pushing to ingester mimir-write-2: user=anonymous: the sample has been rejected because another sample with a more recent timestamp has already been ingested and out-of-order samples are not allowed (err-mimir-sample-out-of-order). The affected sample has timestamp 2026-09-14T13:34:33.879Z and is from series avalanche_gauge_metric_mmmmm_0_207{cycle_id=\"0\", instance=\"test_acbc520f-ad5b-443d-8ffe-61f02c5c564f_aval_aval/0\", job=\"juju_test_acbc520f_aval_prometheus_scrape_aval-0\", juju_application=\"aval\", juju_charm=\"avalanche-k8s\", juju_model=\"test\", juju_model_uuid=\"acbc520f-ad5b-443d-8ffe-61f02c5c564f\", juju_unit=\"aval/0\", label_key_kkkkk_0=\"label_val_vvvvv_0\", label_key_kkkkk_1=\"label_val_vvvvv_1\", label_key_kkkkk_2=\"label_val_vvvvv_2\", label_key_kkkkk_3=\"label_val_vvvvv_3\", label_key_kkkkk_4=\"label_val_vvvvv_4\", label_key_kkkkk_5=\"label_val_vvvvv_5\", label_key_kkkkk_6=\"label_val_vvvvv_6\", label_key_kkkkk_7=\"label_val_vvvvv_7\", label_key_kkkkk_8=\"label_val_vvvvv_8\", label_key_kkkkk_9=\"label_val_vvvvv_9\", series_id=\"1\"} (sampled 1/10)" insight=true

After behaviour

Pack and refresh the coordinator charm. Then, do juju config mimir out_of_order_time_window=5m. The result is that the errors should go away.

Upgrade Notes

@sinapah
sinapah marked this pull request as ready for review September 14, 2026 13:43
@sinapah
sinapah requested a review from a team as a code owner September 14, 2026 13:43
@sinapah
sinapah merged commit b66de98 into track/2.17 Sep 14, 2026
36 checks passed
@sinapah
sinapah deleted the feat/track-2.17/enable-out-of-order-samples branch September 14, 2026 13:56
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Reviewers

@lucabello lucabello lucabello approved these changes

Assignees

No one assigned

Labels

Charm Libraries: Out of Sync

Projects

None yet

Milestone

No milestone

Development

Successfully merging this pull request may close these issues.

3 participants

@sinapah @lucabello @observability-noctua-bot