Inconsistent traffic flow from Grafana to Loki/Alertmanager/Prometheus

[nobuto-m]

When COS Lite is deployed, Grafana will have following data sources. As you see, there is an inconsistency in the traffic flow based on the destination services.

• To Alertmanager: http://192.168.124.138/observability-alertmanager (through Traefik)
• To Loki: http://192.168.124.138/observability-loki-0 (through Traefik)
• To Prometheus: http://prometheus-0.prometheus-endpoints.observability.svc.cluster.local:9090 (through K8s service endpoint)

$ juju status -m observability
Model          Controller          Cloud/Region               Version  SLA          Timestamp
observability  sunbeam-controller  flying-oryx-k8s/localhost  3.6.23   unsupported  05:55:42Z

App           Version  Status  Scale  Charm             Channel           Rev  Address         Exposed  Message
alertmanager  0.27.0   active      1  alertmanager-k8s  1/stable          180  10.152.183.85   no       
catalogue              active      1  catalogue-k8s     1/stable           87  10.152.183.72   no       
grafana       9.5.21   active      1  grafana-k8s       1/stable          160  10.152.183.60   no       
loki          2.9.6    active      1  loki-k8s          1/stable          199  10.152.183.179  no       
prometheus    2.52.0   active      1  prometheus-k8s    1/stable          247  10.152.183.22   no       
traefik       2.11.0   active      1  traefik-k8s       latest/candidate  295  10.152.183.251  no       Serving at http://192.168.124.138

Unit             Workload  Agent  Address     Ports  Message
alertmanager/0*  active    idle   10.1.1.112         
catalogue/0*     active    idle   10.1.2.161         
grafana/0*       active    idle   10.1.2.139         
loki/0*          active    idle   10.1.2.22          
prometheus/0*    active    idle   10.1.1.19          
traefik/0*       active    idle   10.1.1.190         Serving at http://192.168.124.138

Offer                            Application   Charm             Rev  Connected  Endpoint              Interface                Role
alertmanager-karma-dashboard     alertmanager  alertmanager-k8s  180  0/0        karma-dashboard       karma_dashboard          provider
grafana-dashboards               grafana       grafana-k8s       160  2/2        grafana-dashboard     grafana_dashboard        requirer
loki-logging                     loki          loki-k8s          199  2/2        logging               loki_push_api            provider
prometheus-receive-remote-write  prometheus    prometheus-k8s    247  2/2        receive-remote-write  prometheus_remote_write  provider
prometheus-scrape                prometheus    prometheus-k8s    247  0/0        metrics-endpoint      prometheus_scrape        requirer

[nobuto-m]

Relation data:
grafana_show_unit.yaml.gz

[sinapah]

Some notes from my investigation into this issue until it's officially triaged:

As a Grafana datasource Provider, when instantiating the related object, Prometheus declares its "most external" URL as the datasource URL (ref). However, Prometheus also has a _configure() method which handles many hooks. As part of this hook, Prometheus updates the Grafana source URL and explicitly sets it to its internal URL (ref). The associated LoC mentions this issue as the reason for this behaviour.

IIUC, the charm author intentionally made the behaviour such that Prometheus writes its internal URL (AKA the K8s FQDN) as the Grafana datasource URL so that, when Grafana and Prometheus were related within the same model and Prometheus had been signed, then Grafana could just retrieve the CA cert of the authority that signed Prometheus. If Prometheus were to declare its ingressed address though, then Grafana would have to trust Traefik's CA. However, in most COS deployments, Traefik is signed by an external authority, meaning Grafana would have to consume an offer from a CA outside the model in order to be able to trust Traefik. In contrast, COS applications would be signed by an in-model CA.

All of this is to say that, I think, it should be Loki and Alertmanager than advertise their internal URL to Grafana when they have an in-model relation. Now that Ops allows us to determine whether a relation is cross-model or in-model, we can

• write the internal URL if in-model
• write the most external URL if cross-model

as an alternative. However, since COS charms are always signed by an internal CA and Traefik is inside the same model, why not just have all charms write their internal FQDN to Grafana always? This way Grafana can trust only the CA that internally signed Loki, Prometheus, etc.

WDYT @sed-i?

[sed-i]

Yep, sounds right.
I think the discrepancy is mostly for historical reasons.
Now that we have relation-model-get, the endpoint should indeed be sensitive to in-model vs. cross-model.