ModelError: permission denied when setting remote_write endpoint during relation-created

[Abuelodelanada]

Bug Description

When deploying prometheus-k8s alongside other charms, the receive-remote-write-relation-created hook fails with ops.model.ModelError: ERROR permission denied.

The error occurs in PrometheusRemoteWriteProvider._set_endpoint_on_relation because ops internally calls relation-get on the unit's own relation data (to compare before writing), and Juju denies that call.

Traceback excerpt:

 File "lib/charms/prometheus_k8s/v1/prometheus_remote_write.py", line 792, in _set_endpoint_on_relation
     relation.data[self._charm.unit]["remote_write"] = json.dumps(...)
 ops.model.ModelError: ERROR permission denied

To Reproduce

1. Deploy prometheus as part of cos-lite
2. Cross model relate an instance of otelcol in another model to prometheus-receive-remote-write offer
3. Check the following errors in 'juju debug-log'

Environment

Model:       cos
Controller:  ck8s
Cloud:       k8s
Juju:        3.6.19


Applications
App           Version  Status  Scale  Charm                     Channel        Rev  Address         Exposed  Message                                             
────────────  ───────  ──────  ─────  ────────────────────────  ─────────────  ───  ──────────────  ───────  ────────────────────────────────────────────────────
alertmanager  0.28.0   active  1      alertmanager-k8s          2/stable       191  10.152.183.26   no                                                           
ca                     active  1      self-signed-certificates  1/stable       586  10.152.183.118  no                                                           
catalogue              active  1      catalogue-k8s             2/stable       113  10.152.183.218  no                                                           
grafana       12.0.2   active  1      grafana-k8s               2/stable       180  10.152.183.127  no                                                           
loki          2.9.15   active  1      loki-k8s                  2/stable       217  10.152.183.19   no                                                           
prometheus    2.53.3   error   1      prometheus-k8s            2/stable       287  10.152.183.144  no       hook failed: "receive-remote-write-relation-created"
traefik       2.11.0   active  1      traefik-k8s               latest/stable  281  10.152.183.166  no       Serving at http://192.168.1.200                     

Units
Unit            Workload  Agent  Address     Ports  Message                                             
──────────────  ────────  ─────  ──────────  ─────  ────────────────────────────────────────────────────
alertmanager/0  active    idle   10.1.0.249                                                             
ca/0            active    idle   10.1.0.172                                                             
catalogue/0     active    idle   10.1.0.17                                                              
grafana/0       active    idle   10.1.0.33                                                              
loki/0          active    idle   10.1.0.85                                                              
prometheus/0    error     idle   10.1.0.231         hook failed: "receive-remote-write-relation-created"
traefik/0       active    idle   10.1.0.222         Serving at http://192.168.1.200                     

Offers
Offer                            Application   Charm                     Rev  Connected  Endpoint              Interface                Role    
───────────────────────────────  ────────────  ────────────────────────  ───  ─────────  ────────────────────  ───────────────────────  ────────
alertmanager-karma-dashboard     alertmanager  alertmanager-k8s          191  0/0        karma-dashboard       karma_dashboard          provider
certificates                     ca            self-signed-certificates  586  0/0        certificates          tls-certificates         provider
grafana-dashboards               grafana       grafana-k8s               180  0/0        grafana-dashboard     grafana_dashboard        requirer
loki-logging                     loki          loki-k8s                  217  0/0        logging               loki_push_api            provider
prometheus-metrics-endpoint      prometheus    prometheus-k8s            287  0/0        metrics-endpoint      prometheus_scrape        requirer
prometheus-receive-remote-write  prometheus    prometheus-k8s            287  0/1        receive-remote-write  prometheus_remote_write  provider
send-ca-cert                     ca            self-signed-certificates  586  0/0        send-ca-cert          certificate_transfer     provider

Integrations
Provider                            Requirer                     Interface              Type   
──────────────────────────────────  ───────────────────────────  ─────────────────────  ───────
alertmanager:alerting               loki:alertmanager            alertmanager_dispatch  regular
alertmanager:alerting               prometheus:alertmanager      alertmanager_dispatch  regular
alertmanager:grafana-dashboard      grafana:grafana-dashboard    grafana_dashboard      regular
alertmanager:grafana-source         grafana:grafana-source       grafana_datasource     regular
alertmanager:self-metrics-endpoint  prometheus:metrics-endpoint  prometheus_scrape      regular
ca:certificates                     alertmanager:certificates    tls-certificates       regular
ca:certificates                     catalogue:certificates       tls-certificates       regular
ca:certificates                     grafana:certificates         tls-certificates       regular
ca:certificates                     loki:certificates            tls-certificates       regular
ca:certificates                     prometheus:certificates      tls-certificates       regular
ca:send-ca-cert                     traefik:receive-ca-cert      certificate_transfer   regular
catalogue:catalogue                 alertmanager:catalogue       catalogue              regular
catalogue:catalogue                 grafana:catalogue            catalogue              regular
catalogue:catalogue                 prometheus:catalogue         catalogue              regular
grafana:metrics-endpoint            prometheus:metrics-endpoint  prometheus_scrape      regular
loki:grafana-dashboard              grafana:grafana-dashboard    grafana_dashboard      regular
loki:grafana-source                 grafana:grafana-source       grafana_datasource     regular
loki:metrics-endpoint               prometheus:metrics-endpoint  prometheus_scrape      regular
prometheus:grafana-dashboard        grafana:grafana-dashboard    grafana_dashboard      regular
prometheus:grafana-source           grafana:grafana-source       grafana_datasource     regular
traefik:ingress                     alertmanager:ingress         ingress                regular
traefik:ingress                     catalogue:ingress            ingress                regular
traefik:ingress-per-unit            loki:ingress                 ingress_per_unit       regular
traefik:ingress-per-unit            prometheus:ingress           ingress_per_unit       regular
traefik:metrics-endpoint            prometheus:metrics-endpoint  prometheus_scrape      regular
traefik:traefik-route               grafana:ingress              traefik_route          regular

Storage
Unit            Storage ID                Type        Pool        Size   Status    Message                                                                 
──────────────  ────────────────────────  ──────────  ──────────  ─────  ────────  ────────────────────────────────────────────────────────────────────────
alertmanager/0  data/5                    filesystem  kubernetes  1 GiB  attached  Successfully provisioned volume pvc-6af83935-b9aa-40a1-945f-6a63893218dd
grafana/0       database/6                filesystem  kubernetes  1 GiB  attached  Successfully provisioned volume pvc-89bcd55d-7b42-46b3-a5fd-05dd478ead2c
loki/0          active-index-directory/0  filesystem  kubernetes  1 GiB  attached  Successfully provisioned volume pvc-d004a6da-778e-4ab0-b48c-359a08b3d83a
loki/0          loki-chunks/1             filesystem  kubernetes  1 GiB  attached  Successfully provisioned volume pvc-08bc370d-83b8-4f67-a7e5-1fc9da78aec8
prometheus/0    database/2                filesystem  kubernetes  1 GiB  attached  Successfully provisioned volume pvc-dfdef807-cf06-47c7-9514-b1cc57d0c443
traefik/0       configurations/4          filesystem  kubernetes  1 GiB  attached  Successfully provisioned volume pvc-ecfd3292-d9d2-4778-889e-25faf865ba3f

Relevant log output

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/model.py", line 3422, in _run
    result = subprocess.run(args, **kwargs)  # type: ignore
             ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/usr/lib/python3.12/subprocess.py", line 571, in run
    raise CalledProcessError(retcode, process.args,
subprocess.CalledProcessError: Command '('/var/lib/juju/tools/unit-prometheus-0/relation-get', '-r', '35', '-', 'prometheus/0', '--format=json')' returned non-zero exit status 1.

The above exception was the direct cause of the following exception:

Traceback (most recent call last):
  File "/var/lib/juju/agents/unit-prometheus-0/charm/src/charm.py", line 1252, in <module>
    main(PrometheusCharm)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/main.py", line 39, in main
    return _main.main(charm_class=charm_class, use_juju_for_storage=use_juju_for_storage)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 475, in main
    manager.run()
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 459, in run
    self._emit()
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 403, in _emit
    self._emit_charm_event(self.dispatcher.event_name)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/_main.py", line 438, in _emit_charm_event
    event_to_emit.emit(*args, **kwargs)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 351, in emit
    framework._emit(event)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 924, in _emit
    self._reemit(event_path)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/framework.py", line 1030, in _reemit
    custom_handler(event)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/lib/charms/prometheus_k8s/v1/prometheus_remote_write.py", line 760, in _on_consumers_changed
    self.update_endpoint(event.relation)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/lib/charms/prometheus_k8s/v1/prometheus_remote_write.py", line 784, in update_endpoint
    self._set_endpoint_on_relation(relation)
  File "/var/lib/juju/agents/unit-prometheus-0/charm/lib/charms/prometheus_k8s/v1/prometheus_remote_write.py", line 792, in _set_endpoint_on_relation
    relation.data[self._charm.unit]["remote_write"] = json.dumps(
    ~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~~^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/model.py", line 1936, in __setitem__
    self.update({key: value})
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/model.py", line 1976, in update
    if (key not in self and val != '') or (key in self and val != self[key])
        ^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/model.py", line 879, in __contains__
    return key in self._data
                  ^^^^^^^^^^
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/model.py", line 872, in _data
    data = self._lazy_data = self._load()
                             ^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/model.py", line 1838, in _load
    return self._backend.relation_get(self.relation.id, self._entity.name, self._is_app)
           ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/model.py", line 3501, in relation_get
    raw_data_content = self._run(*args, return_output=True, use_json=True)
                       ^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^^
  File "/var/lib/juju/agents/unit-prometheus-0/charm/venv/lib/python3.12/site-packages/ops/model.py", line 3424, in _run
    raise ModelError(e.stderr) from e
ops.model.ModelError: ERROR permission denied

unit-prometheus-0: 02:51:42.442 ERROR juju.worker.uniter.operation hook "receive-remote-write-relation-created" (via hook dispatching script: dispatch) failed: exit status 1

Additional context

No response