We see continuous stream of below messages on grafana-agents trying to stream logs to loki(cos-ha):
2025-12-18T05:27:25Z grafana-agent.grafana-agent[415640]: ts=2025-12-18T05:27:25.039937976Z caller=client.go:419 level=warn component=logs logs_config=log_file_scraper component=client host=10.x.x.x msg="error sending batch, will retry" status=429 tenant= error="server returned HTTP status 429 Too Many Requests (429): Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased, user: 'fake'"
I checked and the config that needs to be tuned would be max_global_streams_per_user (default: 5000) as per the loki official docs but there's no way to change these loki configs from the loki-worker charm it appears.
I also checked the loki config inside the loki-write pod and its shows:
limits_config:
ingestion_burst_size_mb: 15.0
ingestion_rate_mb: 4.0
per_stream_rate_limit: 4MB
per_stream_rate_limit_burst: 15MB
retention_period: 0d
split_queries_by_interval: '0'
We would request to all management of loki configuration from juju charm config in a persistent way.
A quick fix approach would be allow pushing the whole config file at the charm level.
We see continuous stream of below messages on grafana-agents trying to stream logs to loki(cos-ha):
I checked and the config that needs to be tuned would be
max_global_streams_per_user(default: 5000) as per the loki official docs but there's no way to change these loki configs from the loki-worker charm it appears.I also checked the loki config inside the loki-write pod and its shows:
We would request to all management of loki configuration from juju charm config in a persistent way.
A quick fix approach would be allow pushing the whole config file at the charm level.