Redesigning Rules class to support easy addition of new rule types

[swetha1654]

I was looking to add support for Sigma rules in the otlp interface and noticed that
it uses the cosl library, especially the Rules class for rules normalization to a uniform
format, juju topology injection, and validation. Right now, the Rules class is tightly coupled to
Prometheus and Loki and addition of Sigma rules is not straight-forward.

Analysis of the rule format differences

Before I discuss how to restructure the Rules class, here are the key structural differences
between Prometheus / Loki and Sigma rule formats:

Aspect	Prometheus / Loki	Sigma
Grouping	Rules must live inside groups	Each rule is independent, no concept of grouping
Identify	Group name + alert / record name	UUID
Detection logic	expr field that has PromQL/ LogQL queries	detection block with selection and condition
Data source	Implicit (the scrape target)	logsource field (product / category / field)
Validation	Valdiated using cos-tool	Validated using pySigma library
Classification of Rules	Rules can be either RecordingRule or AlertingRule	There is only one format

Analysis of the current state of Rules class

The Rules class has the following public interfaces:

1. add: Add rules from dict to the existing ruleset.
2. add_path: Add rules from a directory.
3. inject_and_validate_rules: Inject Juju topology labels and validate rules using CosTool.
4. as_dict: Return standard rules file in dict representation (used to dump rules into the databag).

Internally, the class does the following:

1. Ingest rules from multiple sources (either add_path / add)
2. Normalize the rules to the "official" format (the rules must have the groups key)
3. Injects Juju topology in:
   1. The group name (Prepends topology identifier + relative path to the group name, appends _rules suffix)
   2. Labels
   3. Expressions (modifies expr to contain the juju topology)
4. Validates the syntax using CosTool.

Questions

1. I noticed that the Juju topology injection happens both when add/add_path and
   inject_and_validate_rules is called. The former is called on the requirer side while the latter is
   used by the provider by injecting the same Juju Topology it receives from the requirer via databag.
   Is there a need to do it both times? It seems a bit redundant to me.
2. Do you see any value in combining add and add_path into a single interface or keep it seperate?

Proposal

I propose the following new structure for Rules - a generic Rules and RuleBackend class.

Rules class

• add(rule_dict, source=None): Parses a rule dict via backend.from_dict (with the instance's
  topology) and appends the results to the internal _items: List[T].
• add_path(dir_path, *, recursive=False): Reads rule files from a directory (or single file),
  builds a RuleSourceInfo from the user, then delegates each file to backend.from_dict.
• as_dict(): Returns the accumulated items in the backend's output format by calling
  backend.as_dict(self._items).
• inject_and_validate(rules, metadata): Reconstructs topology from the metadata dict, runs
  backend.from_dict for injection, then backend.validate for checking. Returns an InjectResult.

RuleBackend class

• file_suffixes (property): Returns the file extensions this backend handles (e.g.
  [".rule", ".yml"] for Prometheus, [".yml", ".yaml"] for Sigma). Used by Rules._from_dir to filter files.
• from_dict(rule_dict, *, source, topology): Takes a raw YAML-loaded dict, normalizes it into
  the backend's internal type T, and injects Juju topology. Returns List[T].
• validate(rules): Takes the serialized output and checks it for correctness. Prometheus delegates to cos-tool; Sigma checks for required fields.
• as_dict(items): Converts List[T] into the backend's output format. Prometheus produces {"groups": [...]}, Sigma produces {"rules": [...]}.

Pros and Cons

Pros:

• Extensibility: Adding a new rule format means writing a new RuleBackend subclass.
• Type safety: Rules[T] ensures that what from_dict produces is what as_dict consumes.

Cons:

• This design also injects topology on both add methods as well as the inject_and_validate
  method. But this can just as easily be removed if we don't see any use case to have it both sides.
• What i don't like is the RuleSourceInfo class leaking into the Sigma backend class as well.
  Currently the public method add exposes group_name and group_name_prefix which are customizable values
  specific to only the Prometheus/Loki backend. Groups are not a concept in Sigma rules. The RuleBackend template
  forces the from_dict method to consume it. Even though the SigmaRuleBackend doesn't have to use it in the method, it will remain in the function definition.

Happy to discuss this further and implement it.

[MichaelThamm]

Answers:

1. The OTLP charmlib is the only user of inject_and_validate. This is important because we want to ensure that the Topology in the databag is used for labeling. This is only a concern for libraries and not charms.
   • Side comment; all additive methods add, add_path, and even inject_and_validate call _from_dict which does the topology injection.
2. add and add_path must remain separate interfaces for backwards compatibility. Also separation of concerns is important

[MichaelThamm]

Your proposal seems relatively sound. However, this will be a significant effort because we need to ensure that we do not break any existing usage of cos-lib. We are the main users of cos-lib, but there may be ad-hoc usages in the Juju ecosystem that we need to be aware of. This would need to coordinated and gated to make sure we do not cause a regression.

[swetha1654]

@MichaelThamm my confusion comes from the side comment you mentioned. The rules already come labelled when they reach otel-collector as the add / add_path calls from_dict (like you mentioned). These methods are called by the OtlpRequirer class. Calling the labelling logic again on rules that are labelled feels redundant.

[MichaelThamm]

Are you referring to the inject_and_validate method? I tend to agree with what you said regarding it being redundant. This is how our other libs were written and I would have to dive deeper into it to determine if we can remove it.

It does seem like we can remove this code from the OTLP lib:

• https://github.com/canonical/charmlibs/blob/main/interfaces/otlp/src/charmlibs/interfaces/otlp/_otlp.py#L493-L498

which would allow us to remove the metadata key from the OTLP lib databag, and the inject_and_validate method from cos-lib.

I created a separate issue for this (Jira) to investigate.

[MichaelThamm]

I just noticed that the architecture diagram you came up with compliments the Sigma module e.g., from sigma.backends.loki import LogQLBackend. I wonder if this will cause friction with the way cos-lib is currently written. I think we could still consider Rules accepting a injector and a validator interface so that we can implement e.g., lokitool, promtool, and sigma-cli as the validators without a large re-write. I would have to see a draft PR to really have a strong opinion

[swetha1654]

Are you referring to the inject_and_validate method?

Yes I am referring to the inject_and_validate method. We don't have to inject juju topology on the provider side since this is already done on the requirer side and the metdata field provided over the otlp interface is not required (unless its used for other purposes).

[swetha1654]

I just noticed that the architecture diagram you came up with compliments the Sigma module

I guess the usage of a RuleBackend class makes it look like it compliements the Sigma module, but i don't expect a large re-write to happen except for the rules.py file. I am working on a POC locally and will reference it in this issue once I feel its mature enough to push it to a fork.

But it would definitely break downstream libraries using Rules class. Do we have a list of interfaces that are using the Rules class @MichaelThamm ?

[swetha1654]

@MichaelThamm here is the updated diagram of the implementation in #192

For your reference, the older Rules class had the following structure:

As you can see, the older Rules class had all the functions in a single place, making it rigid and not easily extensible for other rrule formats. The new structure seperates 2 concerns - user facing APIs in the AbstractRules class, and the validation / injection and other backend logic in the RuleBackend class.

[MichaelThamm]

Closing this since the new approach is captured in this stacked PR:

• feat: Sigma rules pt.3 #197