Loki cluster ring discovery fails (empty ring), after enabling ceph perf counters

[anna-savchenko]

Bug Description

After enabling Ceph performance counters, COS HA Loki cluster starts to misbehave (empty ring state).

Most of the Grafana Dashboards show No Data and throw the below error at different times for different panels:

Error text: unexpected response with status code 500: {"status":"error","errorType":"internal","error":"{\"status\":\"error\",\"errorType\":\"internal\",\"error\":\"expanding series: empty ring\"}"}

I enabled the ceph perf counters on 2026-08-24 at around 13:00. The counters were disabled after approximately one hour.
This can be seen from the dashboard where the perf metrics are used. E.g. Ceph - Cluster:
sum(irate(ceph_osd_op_w_in_bytes{juju_application=~".*",juju_model=~".*",juju_model_uuid=~".*",juju_unit=~".*"}[$__rate_interval]))

(I was able to check the ceph dashboard because the issue comes and goes).

I see the below log starting to appear at exactly 2026-08-24T12:41:** on each node of the cloud where open telemetry collector runs.

ubuntu@storage-04:~$ cat /var/snap/opentelemetry-collector/common/otelcol.log | grep "HTTP 500" | head -n 1
2026-08-24T12:41:57.408Z	info	internal/retry_sender.go:133	Exporting failed. Will retry the request after interval.	{"resource": {"service.instance.id": "6f137075-b172-473c-a609-6bc292e991e8", "service.name": "otelcol", "service.version": "0.130.0"}, "otelcol.component.id": "loki/send-loki-logs/0", "otelcol.component.kind": "exporter", "otelcol.signal": "logs", "error": "HTTP 500 \"Internal Server Error\": at least 2 live replicas required, could only find 1", "interval": "4.794693931s"}

One loki-read unit is blocked:

loki-read/0                 active    idle       10.1.1.225         read ready.
loki-read/1                 blocked   idle       10.1.0.27          node down (see logs)
loki-read/2*                active    idle       10.1.2.21          read ready.

Loki service was down on this unit when I first checked, but now it's up after I started it:

$ juju ssh -m cos --container loki loki-read/1
# pebble services
Service  Startup  Current  Since
loki     enabled  active   3 days ago, at 12:41 UTC

To Reproduce

1. Deploy OpenStack with Ceph
2. Deploy opentelemetry collector
3. Integrate otelcol with ceph-mon
4. Deploy COS HA
5. Integrate otelcol with COS HA
6. Enable ceph perf counters on ceph-mon: sudo ceph config set mgr mgr/prometheus/exclude_perf_counters false
7. Disable the counters after seeing Loki's behavior.

The reason why the ceph perf counters were enabled: https://bugs.launchpad.net/charm-ceph-mon/+bug/2164650

Environment

ceph-mon squid/stable 268
opentelemetry-collector 2/stable 334
loki-worker-k8s 2/stable 99
loki-coordinator-k8s 2/stable 85

Relevant log output

ubuntu@infra-01:~$ juju debug-log -m cos --replay --include loki-read/1 | grep "The worker service appears to be down"
unit-loki-read-1: 12:36:09 ERROR unit.loki-read/1.juju-log The worker service appears to be down and we don't know why. Please check the pebble services' status and their logs.
unit-loki-read-1: 12:50:58 ERROR unit.loki-read/1.juju-log The worker service appears to be down and we don't know why. Please check the pebble services' status and their logs.

ubuntu@infra-01:~$ juju ssh -m cos --container loki loki-read/1
# pebble logs
2026-08-27T13:33:26.492Z [loki] level=error ts=2026-08-27T13:33:26.432998662Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:28.293Z [loki] level=info ts=2026-08-27T13:33:28.261092257Z caller=frontend.go:447 msg="not ready: number of schedulers this worker is connected to is 0"
2026-08-27T13:33:28.493Z [loki] level=error ts=2026-08-27T13:33:28.432919592Z caller=ring_watcher.go:56 component=frontend-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:29.492Z [loki] level=error ts=2026-08-27T13:33:29.433453646Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:32.492Z [loki] level=error ts=2026-08-27T13:33:32.433642069Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:35.493Z [loki] level=error ts=2026-08-27T13:33:35.433147198Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:38.293Z [loki] level=info ts=2026-08-27T13:33:38.261118397Z caller=frontend.go:447 msg="not ready: number of schedulers this worker is connected to is 0"
2026-08-27T13:33:38.492Z [loki] level=info ts=2026-08-27T13:33:38.415639387Z caller=memberlist_client.go:552 msg="initiating cleanup of obsolete entries"
2026-08-27T13:33:38.493Z [loki] level=error ts=2026-08-27T13:33:38.433857867Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:38.493Z [loki] level=error ts=2026-08-27T13:33:38.433903605Z caller=ring_watcher.go:56 component=frontend-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:38.792Z [loki] level=info ts=2026-08-27T13:33:38.704944999Z caller=frontend.go:447 msg="not ready: number of schedulers this worker is connected to is 0"
2026-08-27T13:33:41.493Z [loki] level=error ts=2026-08-27T13:33:41.433123521Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:44.493Z [loki] level=error ts=2026-08-27T13:33:44.433217742Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:47.492Z [loki] level=error ts=2026-08-27T13:33:47.433593266Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:48.293Z [loki] level=info ts=2026-08-27T13:33:48.261252573Z caller=frontend.go:447 msg="not ready: number of schedulers this worker is connected to is 0"
2026-08-27T13:33:48.493Z [loki] level=error ts=2026-08-27T13:33:48.433070996Z caller=ring_watcher.go:56 component=frontend-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:50.492Z [loki] level=error ts=2026-08-27T13:33:50.433457701Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:53.493Z [loki] level=error ts=2026-08-27T13:33:53.433818854Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:56.493Z [loki] level=error ts=2026-08-27T13:33:56.433067064Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:58.293Z [loki] level=info ts=2026-08-27T13:33:58.261130346Z caller=frontend.go:447 msg="not ready: number of schedulers this worker is connected to is 0"
2026-08-27T13:33:58.493Z [loki] level=error ts=2026-08-27T13:33:58.433113848Z caller=ring_watcher.go:56 component=frontend-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:33:59.492Z [loki] level=error ts=2026-08-27T13:33:59.433533106Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:34:02.492Z [loki] level=error ts=2026-08-27T13:34:02.43368224Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:34:05.492Z [loki] level=error ts=2026-08-27T13:34:05.433287876Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:34:08.292Z [loki] level=info ts=2026-08-27T13:34:08.26148241Z caller=frontend.go:447 msg="not ready: number of schedulers this worker is connected to is 0"
2026-08-27T13:34:08.492Z [loki] level=info ts=2026-08-27T13:34:08.414899096Z caller=memberlist_client.go:552 msg="initiating cleanup of obsolete entries"
2026-08-27T13:34:08.492Z [loki] level=error ts=2026-08-27T13:34:08.433086681Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:34:08.492Z [loki] level=error ts=2026-08-27T13:34:08.433137397Z caller=ring_watcher.go:56 component=frontend-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:34:11.493Z [loki] level=error ts=2026-08-27T13:34:11.433804452Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"
2026-08-27T13:34:14.493Z [loki] level=error ts=2026-08-27T13:34:14.432975533Z caller=ring_watcher.go:56 component=querier component=querier-scheduler-worker msg="error getting addresses from ring" err="empty ring"

### Additional context

_No response_

[sed-i]

Hi @anna-savchenko, iiuc, the issue went away after pebble restart and network bandwidth increase?
Is there anything for use to pursue here or can we close the issue?

[anna-savchenko]

Hi @sed-i I didn't find anything else in the logs. I think we close this one.