feat(lib): Remote write rule compression

[MichaelThamm]

Issue

We want compression to avoid hitting databag limits.

Solution

Related:

• chore: Prometheus remote-write schema changes charmlibs#675
• feat: Compressed PRW rules opentelemetry-collector-k8s-operator#370
• feat: Compressed PRW rules mimir-operators#467
• feat: Compressed PRW rules tests opentelemetry-collector-operator#403

Negotiated compression, so no combination of lib versions loses alert rules. The Provider app databag advertises what it can read, consumer picks the best match:

receive-remote-write (provider app data):  alert_rules_encodings: ["lzma", "json"]
send-remote-write    (consumer app data):  alert_rules: <lzma+base64>   # or plain JSON

Consumer	Provider	Result
new	new	compressed
new	old (advertises nothing)	plain JSON, exactly as today
old	new	plain JSON, read fine (reader accepts both)

Consumer: _alert_rules_encoding(relation) inspects the remote app databag; _push_alerts_to_relation_databag encodes accordingly (prometheus_remote_write.py:656). Also pushes alerts on relation_changed, so a relation that started out plain upgrades to compressed once the flag lands.

Provider: _publish_alert_rules_encodings (leader-guarded) called from update_endpoint, relation_changed, plus new leader_elected/upgrade_charm observers so existing relations get the flag after a provider upgrade.

Context

Provider rollback is the one uncovered skew: a downgraded provider can't erase its own alert_rules_encodings key, so the consumer keeps compressing. Recovery is a consumer re-publish after the key is removed manually or a remove integration op.

TODO

• Test this end-to-end with the packed charms from the PRs
• Consider Sina's logic for alert rule validation -- does this require decompression/compression prior?
  • https://github.com/canonical/prometheus-k8s-operator/pull/817/changes
• Backport the charm features

Testing Instructions

With the tandem PRs in Otelcol and Mimir, we should test the compression end-to-end.

Upgrade Mimir, then Otelcol

jd opentelemetry-collector-k8s --trust --channel "dev/edge" otelcol
atelier module add https://github.com/canonical/mimir-operators.git  # dev/edge
jrel otelcol mimir:receive-remote-write
jshu mimir/0 | yq '.[]."relation-info"[] | select(.endpoint == "receive-remote-write").application-data.alert_rules' | jq
# {"groups": [{...}]}
juju refresh mimir --path ./mimir-coordinator-k8s_ubuntu@26.04-amd64.charm \
  --resource nginx-image=ghcr.io/canonical/nginx:1.27.5 --resource nginx-prometheus-exporter-image=ubuntu/nginx-prometheus-exporter@sha256:28fc4e2e0a8e32d08f213889ac5eb613d52f64da47ae5007930adfc688fbbe5c
jshu mimir/0 | yq '.[]."relation-info"[] | select(.endpoint == "receive-remote-write").application-data.alert_rules' | jq
# {"groups": [{...}]}
juju refresh otelcol --path ./opentelemetry-collector-k8s_ubuntu@26.04-amd64.charm \
  --resource opentelemetry-collector-image=ubuntu/opentelemetry-collector@sha256:6df3cad7e93ef008014e85caf66765a69859f4507f1356fdb968f17e68b5766c
jshu mimir/0 | yq '.[]."relation-info"[] | select(.endpoint == "receive-remote-write").application-data'                 
# alert_rules: /Td6WFoAAAT...

❯ jst
Model  Controller  Cloud/Region  Version  SLA          Timestamp
prw    k8s         k8s           3.6.23   unsupported  16:01:29-04:00

App                  Version  Status  Scale  Charm                        Channel   Rev  Address         Exposed  Message
mimir                         active      1  mimir-coordinator-k8s                    1  10.152.183.18   no       
mimir-backend        2.17.10  active      1  mimir-worker-k8s             dev/edge  108  10.152.183.53   no       backend ready.
mimir-read           2.17.10  active      1  mimir-worker-k8s             dev/edge  108  10.152.183.143  no       read ready.
mimir-s3-integrator           active      1  s3-integrator                2/stable  544  10.152.183.139  no       
mimir-write          2.17.10  active      1  mimir-worker-k8s             dev/edge  108  10.152.183.90   no       write ready.
otelcol              0.130.1  active      1  opentelemetry-collector-k8s              0  10.152.183.114  no       

Unit                    Workload  Agent  Address     Ports  Message
mimir-backend/0*        active    idle   10.1.0.240         backend ready.
mimir-read/0*           active    idle   10.1.0.58          read ready.
mimir-s3-integrator/0*  active    idle   10.1.0.80          
mimir-write/0*          active    idle   10.1.0.10          write ready.
mimir/0*                active    idle   10.1.0.225         
otelcol/0*              active    idle   10.1.0.126         

Integration provider                Requirer                          Interface                Type     Message
mimir-s3-integrator:s3-credentials  mimir:s3                          s3                       regular  
mimir-s3-integrator:status-peers    mimir-s3-integrator:status-peers  status_peers             peer     
mimir:mimir-cluster                 mimir-backend:mimir-cluster       mimir_cluster            regular  
mimir:mimir-cluster                 mimir-read:mimir-cluster          mimir_cluster            regular  
mimir:mimir-cluster                 mimir-write:mimir-cluster         mimir_cluster            regular  
mimir:mimir-peers                   mimir:mimir-peers                 mimir_peers              peer     
mimir:receive-remote-write          otelcol:send-remote-write         prometheus_remote_write  regular  
otelcol:peers                       otelcol:peers                     otelcol_replica          peer

Upgrade Otelcol, then Mimir

# same steps as above, deploying Otelcol and Mimir
juju refresh otelcol --path ./opentelemetry-collector-k8s_ubuntu@26.04-amd64.charm \
  --resource opentelemetry-collector-image=ubuntu/opentelemetry-collector@sha256:6df3cad7e93ef008014e85caf66765a69859f4507f1356fdb968f17e68b5766c
jshu mimir/0 | yq '.[]."relation-info"[] | select(.endpoint == "receive-remote-write").application-data.alert_rules' | jq
# {"groups": [{...}]}
juju refresh mimir --path ./mimir-coordinator-k8s_ubuntu@26.04-amd64.charm \
  --resource nginx-image=ghcr.io/canonical/nginx:1.27.5 --resource nginx-prometheus-exporter-image=ubuntu/nginx-prometheus-exporter@sha256:28fc4e2e0a8e32d08f213889ac5eb613d52f64da47ae5007930adfc688fbbe5c
jshu mimir/0 | yq '.[]."relation-info"[] | select(.endpoint == "receive-remote-write").application-data'                 
# alert_rules: /Td6WFoAAAT...

Upgrade Notes

[sed-i]

Recovery is a consumer re-publish after the key is removed manually or a remove integration op.

Should we document in the PR description the manual call to relation-set that fixes a downgrade?