HostDown fires when the host is fine, and doesn’t fire when the host is actually down

[jneo8]

Bug Description

The generic HostDown
alert (expr: up < 1) that comes from generic_alert_groups.application_rules
behaves backwards on setups that push metrics via remote-write instead of being scraped directly by Prometheus. We hit this on the cos-agent path, where an opentelemetry-collector sits next to the exporters, scrapes them on localhost, and remote-writes into COS.

What we actually see:

• When a whole machine goes down, its up series just stops arriving and eventually
  disappears. up < 1 never has anything to match, so HostDown stays quiet and we only get HostMetricsMissing (absent(up)) instead.
• When the machine is perfectly healthy but a single exporter dies, the collector is still alive and keeps pushing up=0 for that one target. That trips up < 1, so we get a critical HostDown even though nothing about the host is down.

So the two alerts end up meaning the opposite of their names.

The reason is that up is only ever produced by whatever is doing the scraping. On the cos-agent path that scraper is the collector living on the same machine as the
exporters, so when the machine dies the collector dies with it and there's nobody left to emit up=0 — the series simply goes absent. A up=0 value can therefore only show up while the host is still alive, which means up < 1 is really "an exporter is unhealthy", not "the host is down". It's only a valid host-down signal under pull-based scraping.

Worth noting cos-lib already knows about this: there's a comment explaining that remote-write setups have no meaningful up and should omit HostDown, and aggregator_rules does exactly that. The problem is the cos-agent path still ships application_rules, which includes HostDown.

To Reproduce

1. Deploy something that ships metrics through the cos-agent path — an
   opentelemetry-collector co-located with the exporters, scraping them locally and
   remote-writing to COS.
2. Power off the machine. You get HostMetricsMissing, but no HostDown.
3. Bring the machine back up and instead stop just one exporter. Now you get
   HostDown, even though the host is up.

Environment

• cos-lib
• opentelemetry-collector
• COS/Prometheus

Relevant log output

...

Additional context

This came out of canonical/hardware-observer-operator#536, which describes the exact
same thing:

if the entire machine goes down we get HostMetricsMissing but we don't get HostDown

but then we get HostDown when smartctl exporter stops working but the machine is up

Ideally we'd have a generic ruleset for remote-write consumers that drops HostDown
(up < 1) and leans on absent(up) instead, with a way for those consumers to select it — and it'd help to spell out in the docs which ruleset is meant for pull vs push.