Skip to content

loki.source.windowsevent: never re-subscribes after "The specified channel could not be found" — whole component stops shipping until service restart #7060

Description

@yotake

What's wrong?

loki.source.windowsevent never recovers after its Windows event subscription breaks. Once EvtNext/FetchEvents returns The specified channel could not be found, the component logs

level=error msg="error fetching events" component_path=/ component_id=loki.source.windowsevent.default err="The specified channel could not be found."

every poll_interval (3 s) forever, the service stays Running, loki.write keeps returning 204 for the other components, and no events from any channel covered by that component are shipped again until the Alloy service is restarted. With one component whose xpath_query spans ~30 channels (Security, System, Application, Microsoft-Windows-Windows Defender/Operational, …) a single transiently unregistered channel silently stops the whole block.

Observed on 13 machines (Windows Server 2022 and 2025, Alloy v1.18.1):

  • 2026-09-02/04: started within seconds of the Microsoft Defender antimalware platform update (KB4052623) on every machine.
  • 2026-09-07: recurred on 5 machines within a 90-minute window with no Defender platform update and no .evtx re-created; trigger unknown. Channels referenced by the query all exist again when checked later, so the condition is transient — only the subscription handle is dead.

Looking at internal/component/loki/source/windowsevent/target.go, the fetch loop logs the error and breaks; there is no re-subscribe on ERROR_EVT_CHANNEL_NOT_FOUND (15007) or on other subscription errors, so the same dead handle is polled again on the next tick. The CHANGELOG through v1.19.2 has no related change.

Expected behaviour

When FetchEvents fails with a channel/publisher error, the component should close the subscription and re-subscribe from the bookmark (optionally with backoff), the way Winlogbeat does since elastic/beats#34605 ("Gracefully handle event channel not found errors"). At minimum the failure should be surfaced as a component health state (unhealthy) rather than only as a repeating error log line, so operators can alert on it.

Steps to reproduce

  1. Configure one loki.source.windowsevent with an xpath_query covering several channels, including one owned by a provider that gets re-registered by an update (e.g. Microsoft-Windows-Windows Defender/Operational).
  2. Apply a Defender antimalware platform update (or otherwise unregister and re-register the channel).
  3. Observe the repeating error fetching events line and the absence of new entries from all channels of that component in Loki. Restart-Service Alloy fixes it (bookmark resend works).

System information

Windows Server 2022 Datacenter (Hyper-V hosts) and Windows Server 2025 (guests), x64

Software version

Grafana Alloy v1.18.1 (Windows service). Not fixed in v1.19.2 per CHANGELOG.

Configuration

loki.source.windowsevent "default" {
  xpath_query = "<QueryList><Query Id=\"0\"><Select Path=\"Security\">*</Select><Select Path=\"System\">*</Select><Select Path=\"Application\">*</Select><Select Path=\"Microsoft-Windows-Windows Defender/Operational\">*</Select> ... ~28 more channels ... </Query></QueryList>"
  use_incoming_timestamp = true
  forward_to = [loki.write.default.receiver]
}

Logs

ts=2026-09-08T15:03:44.0592303Z level=error msg="error fetching events" component_path=/ component_id=loki.source.windowsevent.default err="The specified channel could not be found."
ts=2026-09-08T15:03:41.0596267Z level=error msg="error fetching events" component_path=/ component_id=loki.source.windowsevent.default err="The specified channel could not be found."

(repeats every 3 s for ~30 hours until the service was restarted)

Workaround

An external watchdog (scheduled task) that counts these error events in the Application log and restarts the Alloy service, plus splitting the channels into several smaller loki.source.windowsevent components to limit the blast radius.

Activity

Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

No one assigned

    Labels

    No labels
    No labels

    Type

    No type

    Projects

    No projects

      Milestone

      No milestone

      Relationships

      None yet

      Development

      No branches or pull requests

      Issue actions