fix(grug): the enforcement gauge reports the whole fleet, so the monitor cannot latch - #815
Merged
Merged
Conversation
The monitor filtered on the `enforcement_type:none` TAG. A repo that gained enforcement stopped matching that tag, so instead of reporting a healthy value into its multi-alert group, the group went SILENT - and Datadog holds a silent metric group in its last state for 24h. The alert stayed red long after the gap was gone. Measured live 2026-08-01: yuzu-yard-sale emitted its last point at 04:00:00Z; at 07:18:15Z the monitor still read Alert with ZERO repos in the gap, past 3x its own `last_1h` window. `last_triggered_ts` was 2026-07-28, 3.5 days stale. `group_retention_duration` cannot shorten that: it is documented as APM/Audit/CI/Error-Tracking/Event/Logs/RUM only, not `metric alert`. So threshold on the VALUE instead, which already encodes the state (grug_managed 1.0, external 0.5, none 0.0, error -1.0). Every repo keeps reporting into its own group, and a repo that gains enforcement flips 0.0 -> 0.5/1.0 and RESOLVES rather than going quiet. This also closes a blind spot that was worse than the latch. Under the tag filter, an auth or rate-limit outage emitted `enforcement_type:error` for every repo and `none` for none of them - so the gap monitor went QUIET during exactly the incident that made enforcement unknowable. `error` is -1.0, so it now trips the threshold. The monitor is renamed "not provably gated" because that is what it now means. Verified against live data: the new query puts 16 repos in scope (14 at 0.5, 2 at 1.0) and NONE breach - so this also clears the current red. Residual and inherent to Datadog: a repo leaving the fleet entirely (tpm_enabled=false, archive, uninstall) stops emitting and its group still ages out on the 24h default. Stated in the monitor message so the next operator does not chase it. Refs #716
Follow-up to 974e024 on the same issue, after review found that commit's central claim did not survive its own evidence. 974e024 blamed the tag filter and cited yuzu-yard-sale's 04:00Z->07:18Z red-with-no-gap window as proof. But yuzu left via `tpm_enabled=false`, which `poller_handler` handled with a bare `continue` - so it stopped emitting ENTIRELY. That is a different latch from the one the commit fixed, and it is the one the measurement actually proves. The tag-filter latch is real by inspection but has never been observed. Both are the same shape: a repo that stops reporting keeps its last verdict, because Datadog holds a silent multi-alert group for 24h and `group_retention_duration` is not available for `metric alert`. So fix the cause rather than the symptom: EVERY visible repo reports every cycle. An opted-out repo emits `opted_out` (1.0) instead of being skipped, which resolves its group on the next evaluation. It is still not detected - opting out should not spend a GitHub API call. `force_disable_enforcement` now counts as an opt-out too. CONTEXT.md documents it as one, but the poller only checked `tpm_enabled`, so a repo using the documented escape hatch kept detecting `none` and pinned this monitor red permanently. Review also caught real drift, all fixed here: - DESIGN.md asserted -1.0 in one row while the row above said the values were 1.0/0.5/0.0 only. - CONTEXT.md never learned `error`, `opted_out` or "provably gated", though the monitor name and query now depend on all three. - The postmortem narrative was restated in four places and the value map in five. Both now live once: ADR-0022 and `emit_enforcement_metric`. - The monitor message had grown into a mini-runbook and told operators to use `tpm_enabled=false` while CONTEXT.md documents `force_disable_enforcement`. Trimmed to the sibling shape. Suites: 1872 webhook, 288 api, 24 infra. Refs #716
quadseven
force-pushed
the
fix/716-enforcement-monitor-latch
branch
from
August 2, 2026 14:47
474d9ab to
7848a35
Compare
Owner
Author
|
Force-pushed to drop This branch is now the enforcement-gauge work only - 8 files, no |
Grug walk most of trail. Some ground not walked.fix(grug): the enforcement gauge reports the whole fleet, so the monitor cannot latch Grug Elder - Partial coverage - no markings on the ground Grug walked - check `neutral`
Some ground not walked this pass - part of the diff did not fit one look. What Grug did walk is above. Grug not say trail safe for ground Grug not walk. |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Why
The enforcement-gap monitor latched. On 2026-08-01 it read
Alertfor over three hours while zero repos were in the gap:Datadog holds a silent multi-alert group in its last state for 24 hours, and
group_retention_durationcannot shorten that - it is documented for APM/Audit/CI/Error-Tracking/Event/Logs/RUM monitors, notmetric alert. So a repo that was red when it stopped reporting stayed red.Full write-up, including which cause is observed and which is inferred: ADR-0022.
Summary
Every repo the installation can see reports a value on every poll cycle. Leaving the gauge is never how a repo stops alerting; reporting a healthy value is.
opted_out(1.0) instead of being skipped, so their group resolves on the next evaluation.force_disable_enforcementis documented in CONTEXT.md as an enforcement opt-out, but the poller only ever checkedtpm_enabled- so a repo using the documented escape hatch kept detectingnoneand pinned this monitor red permanently. That is a latent bug this PR also fixes.< 0.5), never on theenforcement_typetag, so no state transition can make a series vanish.observability.emit_enforcement_metricowns the value map; nothing else restates it.Acceptance criteria
test_enforcement_pass_emits_opted_out_instead_of_dropping_the_repo, which also asserts it costs no GitHub API callforce_disable_enforcementcounts as an opt-out -test_enforcement_pass_treats_force_disable_as_an_opt_outtest_enforcement_gap_query_thresholds_on_value_not_the_state_tagSize
Size: M
Out of scope
pup monitors getreturnsstate: nullwithoverall_state_modifiedfrozen at creation; live status only comes frompup monitors search. That is a vendor/tooling property, not fixable from grug's Pulumi. This is why the issue isRefs, notcloses.docs/RUNBOOK.md#enforcement-gapanchor - that file has no enforcement section at all. Pre-existing; flagged rather than silently expanded.Reviewer note
A first attempt (
974e024) blamed only the tag filter and cited the yuzu measurement as proof. Review caught that yuzu left viatpm_enabled=false, which is a different latch - so the evidence did not support the claim.7848a35fixes the actual cause and ADR-0022 is explicit about which cause is observed and which is inferred.Refs #716