Remove HostMetricsMissing from charms that don't remote write into Prometheus

[sinapah]

Bug Description

Charms such as Grafana and AM, for example, are scraped by Prometheus for metrics. Unlike Otel Collector, they do not remote write into Prometheus. In the generic alert rules we define in cosl, we define HostDown and HostMetricsMissing which are supposed to alert when a scraped host is down and when a remote writing host is down respectively.

cos-lib/src/cosl/rules.py

Lines 98 to 127 in 32af2d0

	_generic_alert_rules: Final = SimpleNamespace(
	# We use "5m" to avoid false positives on expected temporary "down", e.g. during intentional (re)start.
	# Juju topology will be later injected by providers of alert rules.
	host_down={
	"alert": "HostDown",
	"expr": "up < 1",
	"for": "5m",
	"labels": {"severity": "critical"},
	"annotations": {
	"summary": "Host '{{ $labels.instance }}' is down.",
	"description": """Host '{{ $labels.instance }}' is down, failed to scrape.
	VALUE = {{ $value }}
	LABELS = {{ $labels }}""",
	},
	},
	host_metrics_missing={
	"alert": "HostMetricsMissing",
	# We use "absent(up)" with "for: 5m" because the alert transitions from "Pending" to "Firing".
	# If query portability is desired, "absent_over_time(up[5m])" is an alternative.
	"expr": "absent(up)",
	"for": "5m",
	"labels": {"severity": "critical"},
	"annotations": {
	"summary": "Metrics not received from host '{{ $labels.instance }}', failed to remote write.",
	"description": """Metrics not received from host '{{ $labels.instance }}', failed to remote write.
	VALUE = {{ $value }}
	LABELS = {{ $labels }}""",
	},
	},
	)

For Otel Collector, which remote writes into Prometheus, we do not have use HostDown. Likewise, for AM and Grafana, which are scraped, we should not use HostMetricsMissing. As we say [in the docs](This alert is only applicable to clients (aggregators) that push metrics via remote-write, such as grafana-agent or opentelemetry-collector.)

This alert (HostMetricsMissing) is only applicable to clients (aggregators) that push metrics via remote-write, such as grafana-agent or opentelemetry-collector.

To Reproduce

N/A

Environment

N/A

Relevant log output

N/A

Additional context

No response

[sinapah]

To continue:

• HostMetricsMissing is useful for these charms (meaning the ones that are scraped by Prometheus) only when they are scraped by Grafana Agent and have their metrics remote written. In these cases, we should only use the HostMetricsMissing alerts and not HostDown.