Validate configuration before and after building

[lucabello]

Enhancement Proposal

The Mimir configuration we put in relation data should be validated twice:

• when the coordinator builds it, via a Pydantic model that maps to the Mimir configuration
• when the worker receives it (and before trying to start), using mimirtool runtime-config to avoid the "retry loop" on a faulty configuration file

This should resolve retrying for a very long time on a faulty config, while the correct one is in a pending event. This might also be the cause of flakiness in integration tests.

[Moetez-Fradi]

Hello @lucabello , I want to work on this if nobody is laready working on it. I'm a SWE student and here is what I understood and what I thought of as plan (I'm very open to discuss it)

for now, coordinator/src/mimir_config.py builds the config as a plain dict and does yaml.dump(...) with no validation, and on the worker side a faulty config should lead to the retry loop (SERVICE_START_RETRY_STOP = stop_after_delay(120)), which I think is a source of the integration-test flakiness.

The plan ( potentially 2 different PRs ):

Coordinator-side (Pydantic)

 * Add Pydantic models covering the config sections the charm actually produces (alertmanager, ingester, ruler etc.), rather than modelling all of Mimir's options.

 * Validate the assembled config in MimirConfig.config() before yaml.dump, so a malformed config fails fast on the coordinator with a clear log instead of being shipped over relation data.

 * Add unit test(s) in coordinator/tests/unit/test_mimir_config.py.

Worker-side validation (before start)

 * Validate the written config file inside the workload container (via Pebble exec) using the mimirtool before starting the service.

 * On failure, set a blocked status with a helpful message instead of entering the long retry loop.

I also have some questions that could help:

for the first part, should the Pydantic model be strict (faill hard) or just give warnings ? and when should validation errors surface ?
For step 2, is mimirtool available in the worker container ?
and finally, should it be one or 2 separate PRs ?

I'll start working as early as possible, thank you !

[MichaelThamm]

I think the first step would be to build a minimal (only what we are using - not the full config) Pydantic model of the upstream Mimir config with the same style as done in Tempo:

• https://github.com/canonical/tempo-operators/blob/main/coordinator/src/tempo_config.py

Following this Pydantic model feature, we can then focus on when/where to validate. Some things to consider are:

1. Corner cases like upgrade paths across the Mimir charm tracks e.g., 2.17 -> dev.
2. What would happen if a new field was added in dev that 2.17 did not have, or vice versa.

We need to make sure that any failed validation (e.g., ExtraValues) is handled gracefully.

[Moetez-Fradi]

I'll get to it right now, thank you for your guidance !

[Moetez-Fradi]

Hello @MichaelThamm @lucabello,

I opened the first PR as discussed (minimal Pydantic model of the config sections we already emit in tempo-style)

PR link: #461

AFter this one is validated, I'll work on where to surface validation errors and on how to avoid the retry loop

Happy to adjust the model or followup PRs based on your feedback!