Skip to content

Coroot alerts can outlive their application report #5594

Closed

Description

@devantler

🤖 Generated by the Agentic Engineer

Expected Behavior

Alert state should distinguish a check that no longer applies to an application from a check whose telemetry is temporarily unavailable. Retiring an inapplicable report should not leave an unexplained firing alert, and loss of monitoring must remain visible.

Actual Behavior

At Coroot source 15137ef2f43e01d0a87245e44b937b25515e0afe, the evaluation loop skips an application when the selected check has no report. The retirement pass retains the existing alert while its rule still matches and the application exists. A fresh application view can therefore have no corresponding report while its stored alert remains firing. Current normal-SSO views reproduce that inconsistency; detailed operator evidence remains private.

This is a source-traced lifecycle case, not a completed native regression or proof of the original metric-attribution cause. The UI does not expose the stored alert's last evaluation timestamp. A missing availability check within an UNKNOWN database report can mean lost instrumentation and must not be treated as recovery.

Steps to Replicate

  1. Use one matching enabled availability rule and a firing alert for an application, with no pending or keep-firing delay.
  2. Audit a later world where the same live application no longer has the associated report, and run the normal alert evaluation and retirement passes.
  3. Compare that retained alert with a control whose check is explicitly healthy, and with a known database whose telemetry is missing.

Acceptance Criteria

  • Reproduce the lifecycle with the existing native database and watcher APIs, including the explicit-healthy control.
  • Define and test a positive boundary between retired applicability and missing instrumentation, including UNKNOWN reports, empty instances and collector outages.
  • Repair the owning source, validate its complete required checks and review, and adopt a verified publication through the protected platform delivery path.
  • Observe the obsolete alert retire naturally while genuine database failures and monitoring gaps remain visible, without dismissal, suppression or a changed observation window.

Planning

Bug; medium priority, high end-to-end effort, Standard service class. The maintainer selected stabilization work on 3 November 2026 and explicitly waived the active WIP ceiling for this work. No active competing claim or implementation was found before selection.

The first bounded delivery is a native database/watcher regression and repair for treating an explicit UNKNOWN check as recovery. Source inspection shows that loss of telemetry can produce UNKNOWN for an existing check; this must preserve a firing or suppressed alert and restart any pending continuous-breach timer. Genuine OK recovery and explicit rule/app retirement must retain their existing behavior. Separately, absent PostgreSQL reports still need a positive applicability boundary; this first repair does not claim to retire those alerts.

First started: 2026-11-03. Acceptance, original scope and deployment verification remain open.

The alert lifecycle belongs to Coroot. An exclusion or dismissal inside the platform cannot correct that lifecycle and would weaken the observation instead. A minimal upstream report or fix is justified after a synthetic native reproduction, using only public source and synthetic data. Coroot is a public platform dependency unrelated to employment; its current contribution policy contains no AI-assistance prohibition. The UNKNOWN-state correction is now proposed in coroot/coroot#1099, offered for upstream review after local validation and passing hosted Go/UI checks. The absent-report applicability boundary and runtime acceptance remain open.

Part of #2043.

Activity

  1. self-assigned this
    on Oct 29, 2026
  2. added theissue type on Oct 29, 2026
  3. moved this to 📥 Backlog in 🌊 Project Boardon Oct 29, 2026
  4. updated Priority Medium and Effort High on Oct 29, 2026
  5. added a commit that references this issue on Nov 3, 2026
  6. moved this from 📥 Backlog to 🫴 Ready in 🌊 Project Boardon Nov 3, 2026
  7. moved this from 🫴 Ready to 🏃🏻‍♂️ In Progress in 🌊 Project Boardon Nov 3, 2026
  8. added 3 commits that reference this issue on Nov 3, 2026
  9. devantler commented on Nov 12, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Rollout check after #5968 (read-only, 2026-11-12 ~12:20 UTC). The first slice is live. The case this issue is about still happens.

    • The fix is running. The production Coroot server runs exactly the signed build that fix(coroot): deploy the alert observation repair #5968 pinned (v1.28.1-checkpoints.2-900535fe…, digest sha256:0f77bc49…), with both replicas ready.
    • The old alerts have not cleared. Four postgres-availability alerts are still firing on applications that run no PostgreSQL: two monitoring components, one cluster component and one tenant application. They opened between 29 October and 3 November. None has been updated since then: the newest update is 3 November 14:20 UTC, more than two days before the rollout. The other firing alerts kept updating as normal through this check.

    What this means: #5844/#5968 delivered the part about missing data (an alert whose data is missing stays visible). They were never meant to retire an alert whose report no longer applies, and that is what these four alerts show. So the second and fourth acceptance criteria are still open, and these four alerts are the live examples for them. Nothing was dismissed or resolved by hand; doing that would hide the lifecycle this issue needs to fix.

    Next slice: the boundary between "this report no longer applies" and "the data is missing" (acceptance criterion 2), built as a native regression in Coroot, then sent upstream as the issue plans. After that fix is deployed, these four alerts should retire on their own. That is how the fourth criterion gets checked.

  10. added a commit that references this issue on Nov 12, 2026
  11. devantler commented on Nov 12, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Second slice in review: #6129 (acceptance criterion 2). Our patched Coroot build now retires a check alert only when there is positive proof its report no longer applies: the alert records which report it came from, a fresh audit no longer creates that report at all, and the application has an instance observed running on a node whose agent is reporting. Native regressions against the pinned Coroot source cover the retirement (active and suppressed) and five missing-data cases that must keep firing: no instances, a collector outage on the node, an unobserved container, a database whose instrumentation is missing (UNKNOWN report), and an alert with no recorded report. Removing any one guard makes its case fail.

    Remaining after merge: the publish run builds the new image from main; a follow-up pins its digest in production (criterion 3), then the four stuck alerts should retire on their own (criterion 4).

  12. devantler commented on Nov 12, 2026

    @devantler
    ContributorAuthor

    🤖 Generated by the Agentic Engineer

    Rollout check after #6131 (read-only, 2026-11-12 ~16:40 UTC). All four acceptance criteria now hold, so this issue is done.

    • The fix is running. Both production Coroot server replicas run the build fix(coroot): deploy the inapplicable-report alert retirement #6131 pinned, v1.28.1-checkpoints.3 at digest sha256:0f9ee31a…. They restarted at about 16:07 UTC, while fix(coroot): deploy the inapplicable-report alert retirement #6131 was in the merge queue.
    • The obsolete alerts retired on their own. The four postgres-availability alerts that had stayed open with no matching report closed at 16:08:18 UTC, on the first evaluation after the restart. Coroot's own record shows manually_resolved_at: null for all four, so nobody dismissed, suppressed or resolved them by hand. No observation window was changed.
    • Real failures stay visible. Firing alerts of other kinds kept firing through the rollout: storage, instance availability, DNS and Kubernetes events. No real PostgreSQL outage happened during the window, so the "genuine database failure keeps firing" case is shown by the native regression cases from fix(coroot): retire check alerts whose report no longer applies #6129, not by a live outage. Those cases cover a database report going UNKNOWN while its type is still detected, a JVM report that disappears with its own instrumentation, and a retired alert that fails again.
    Criterion Delivered by
    1. Reproduce with native database and watcher APIs, with an explicit-healthy control #5968
    2. A positive boundary between "no longer applies" and "instrumentation lost" #6129
    3. Repair the source, validate, publish through the protected delivery path #6129 (source and image) + #6131 (pin)
    4. The obsolete alert retires naturally while real failures stay visible this check

    The upstream proposal coroot/coroot#1099 is still open. That is the upstream contribution path, not one of this issue's criteria. If it merges, the platform can drop its patch in a later dependency update.

  13. moved this from 🏃🏻‍♂️ In Progress to ✅ Done in 🌊 Project Boardon Nov 12, 2026
Sign up for free to join this conversation on GitHub. Already have an account? Sign in to comment

Metadata

Metadata

Assignees

Labels

No labels
No labels

Type

Fields

Priority

Medium

Effort

High

Projects

Milestone

No milestone

Relationships

None yet

Development

No branches or pull requests

Issue actions