docs/loki/

[giscus[bot]]

docs/loki/

How to deploy Grafana Loki in our Raspberry Pi Kuberentes cluster.

https://picluster.ricsanfre.com/docs/loki/

[i5Js]

The new helm chart requires an schema to install loki, otherwise you will get an error:

Error: execution error at (loki/templates/validate.yaml:40:4): You must provide a schema_config for Loki, one is not provided as this will be individual for every Loki cluster. See https://grafana.com/docs/loki/latest/operations/storage/schema/ for schema information. For quick testing (with no persistence) add `--set loki.useTestSchema=true`
helm.go:84: [debug] execution error at (loki/templates/validate.yaml:40:4): You must provide a schema_config for Loki, one is not provided as this will be individual for every Loki cluster. See https://grafana.com/docs/loki/latest/operations/storage/schema/ for schema information. For quick testing (with no persistence) add `--set loki.useTestSchema=true`

[i5Js]

It can be fixed if we add to the values the following:

  #schema config
  useTestSchema: true
  testSchemaConfig:
    configs:
      - from: 2024-04-01
        store: tsdb
        object_store: filesystem
        schema: v13
        index:
          prefix: index_
          period: 24h

However, and I can't remeber if it happened before, there is one pod in pending:

pod/loki-chunks-cache-0 0/8 nodes are available: 3 node(s) had untolerated taint {node-role.kubernetes.io/control-plane: }, 5 Insufficient memory. preemption: 0/8 nodes are available: 3 Preemption is not helpful for scheduling, 5 No preemption victims found for incoming pod.

I don't know how much it needs.

[ricsanfre]

Hi Javi

It seems that major new release of loki (3.0), linked to major helm chart releases 6.x, is causing some issues. They have removed the default configuration for the schema_config that previously was by default using boltDb and now they are moving to use TSDB, but not providing a default value for schema_config in values.yaml. The default installation is pushing to use a testSchema, but by default the use of that test schema is not set to true. For production environment, not testing a valid schema_config should be provided.

Anyway the Loki release and its helm chart have lots of issues. Community is tracking those issues in: grafana/loki#12506

I will open an issue in the project repository to track it.
Thanks

Regards
Ricardo

[ricsanfre]

Hi,
I have updated installation procedure so latest Loki release can be installed.

Regards
Ricardo

[ilbarone87]

@ricsanfre I've followed your guide to deploy loki but after upgrading from 2.x version i'm getting a lot of these errors

level=error ts=2024-05-16T20:57:24.847453854Z caller=index_set.go:310 table-name=loki_index_19815 msg=\"sync failed, retrying it\" err=\"invalid tsdb path: /var/loki/tsdb-shipper-cache/loki_index_19815/compactor-1712709193.r\""

Did you experience the same?

[ilbarone87]

Also have noticed a massive increase in log counts

You can spot the moment when i merged the PR to update to loki 3.x

[ilbarone87]

Seems that main culprit is Loki-canary that got created after the upgrade of the chart

[ilbarone87]

Maybe this grafana/loki#12765

[ricsanfre]

Hi @ilbarone87, I am experiencing same type of logs.

level=error ts=2024-05-18T06:47:45.085658797Z caller=index_set.go:310 table-name=loki_index_19861 user-id=fake msg="sync failed, retrying it" err="open /var/loki/tsdb-shipper-cache/loki_index_19861/fake/1716014855853323220-compactor-1715428847000-1716014449612-c100f9a2.tsdb: no such file or directory"
level=error ts=2024-05-18T06:47:45.099827152Z caller=index_set.go:310 table-name=loki_index_19861 user-id=fake msg="sync failed, retrying it" err="open /var/loki/tsdb-shipper-cache/loki_index_19861/fake/1716014855853323220-compactor-1715428847000-1716014449612-c100f9a2.tsdb: no such file or directory"
level=error ts=2024-05-18T06:47:45.099922965Z caller=table_manager.go:135 index-store=tsdb-2024-04-01 msg="error syncing local boltdb files with storage" err="failed to sync index set fake for table loki_index_19861: open /var/loki/tsdb-shipper-cache/loki_index_19861/fake/1716014855853323220-compactor-1715428847000-1716014449612-c100f9a2.tsdb: no such file or directory"

This error is related to grafana/loki#11387

The problem is that during the upgrade form 2.x to 3.x storage schema has changed from boltdb to tsdb. The old schema_config using boltdb were removed from the configuration but there were already indices following boltdb schema, which are the ones that now are failing to synchronize because loki does not have the proper configuration.

See migration procedure here https://grafana.com/docs/loki/next/setup/migrate/migrate-to-tsdb/
The configuration of pre-migration boltdb schema should have been kept in the system during the migration and specify one day in the future, when new tsbd schema were going to be used.

schema_config:
  configs:
    - from: 2023-01-01
      store: boltdb-shipper
      object_store: filesystem
      schema: v11
      index:
        prefix: index_
        period: 24h
    - from: 2024-06-01
      store: tsdb
      object_store: filesystem
      schema: v13
      index:
        prefix: index_
        period: 24h

I suppose those errors will disappear when those old boltdb indices are purged from the system (retention data policy)

[sbatururimi]

what is the best dashboard to be used with Loki logs? I didn't get the whole picture tbh :(

[Guillembonet]

Hey, first of all, your pi cluster guide is great, I wish I found it before I set up my cluster as I would have saved a lot of time, but no regrets because it was also nice to learn. I ended up with a similar setup.
I would like to know how do you manage log retention. I think that by default Loki just keeps logs forever unless you do something in the config or in the s3 object policy, but it looked like you didn't configure anything.

[ricsanfre]

Thanks Guillem.

No, currently I do not have any retention policy configured in Loki. It seems that Loki's Compactor can be configured to do so: https://grafana.com/docs/loki/latest/operations/storage/retention/, but I do not have this component configured in my Loki set-up.
Are you using a "SimpleScalable" Loki deployment as I do? Maybe you can help me to add that configuration to my cluster and the project documentation. Any PR to this project is more than welcome.

[Guillembonet]

Cool, I'll try to make a PR at some point Yes, I have the same setup as you and I currently have this in my Loki config:

         compactor:
            delete_request_store: s3
            retention_enabled: true

          limits_config:
            allow_structured_metadata: true
            volume_enabled: true
            max_query_lookback: 15d
            retention_period: 15d

But I didn't check if it's working fine. Once I do I'll try to make a PR!