Skip to content

Expose loki configurations from charm config #22

Description

@canonikamal

We see continuous stream of below messages on grafana-agents trying to stream logs to loki(cos-ha):

2025-12-18T05:27:25Z grafana-agent.grafana-agent[415640]: ts=2025-12-18T05:27:25.039937976Z caller=client.go:419 level=warn component=logs logs_config=log_file_scraper component=client host=10.x.x.x msg="error sending batch, will retry" status=429 tenant= error="server returned HTTP status 429 Too Many Requests (429): Maximum active stream limit exceeded, reduce the number of active streams (reduce labels or reduce label values), or contact your Loki administrator to see if the limit can be increased, user: 'fake'"

I checked and the config that needs to be tuned would be max_global_streams_per_user (default: 5000) as per the loki official docs but there's no way to change these loki configs from the loki-worker charm it appears.

I also checked the loki config inside the loki-write pod and its shows:

limits_config:
  ingestion_burst_size_mb: 15.0
  ingestion_rate_mb: 4.0
  per_stream_rate_limit: 4MB
  per_stream_rate_limit_burst: 15MB
  retention_period: 0d
  split_queries_by_interval: '0'

We would request to all management of loki configuration from juju charm config in a persistent way.
A quick fix approach would be allow pushing the whole config file at the charm level.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions