Repository navigation
fix: require sustained threshold before alerting on CPU and memory (#66) - #68
Merged
Merged
Conversation
The CPU and memory rules fired on the first sample over threshold. At the default 15s interval a cron job, backup, package upgrade or build that fills one sample window sent "above threshold" and then "recovered" a sample later: two pushes for something that is not an incident. The engine already supports a consecutive-sample sustain (added for temperature in #25); the system rules now use it. The window is defined as a duration (one minute) and converted to samples from sample_interval, so a shorter interval does not shrink it. Because CPU is an interval average, N violating samples mean at least N x interval of real saturation. A burst shorter than the window now sends neither an alert nor a recovery notice. Recovery, cooldown and snooze behave as before. No config key is added.
|
Important
This repository does not receive automatic reviews because it has fewer than 10 stars. ⚙️ Run configurationConfiguration used: Path: .coderabbit.yaml Review profile: CHILL Plan: Advanced Run ID: Thanks for using CodeRabbit! It's free for OSS, and your support helps us grow. If you like it, consider giving us a shout-out. Comment |
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
What & why
Fixes #66.
The
cpuandmemrules inevalSystemfired on the first sample over threshold. At the default 15ssample_interval, any cron job, backup,aptupgrade or build that fills one sample window produced an "above threshold" push followed by a "recovered" push 15s later. The 10-minute cooldown doesn't prevent that pair, because it only blocks a second fire.The alert engine already supports a consecutive-sample sustain (
ThresholdOpts.Sustain, added for temperature in #25). This PR makes the system rules use it:systemSustain = time.Minuteconstant and asustainSamples(d, interval)helper ininternal/agent/agent.gothat converts the window to a sample count (ceil, min 1).cpuandmempassSustaincomputed from the configured interval. Disk and battery are unchanged.Behavior after this change:
Trade-off: a real CPU/memory incident is reported about a minute later, and a load oscillating inside the hysteresis band (e.g. 85↔95%) doesn't build a streak. Both match the existing temperature semantics.
Also updated the outdated
tempSustainSamplescomment,docs/ARCHITECTURE.md(Sustain bullet), and the README "Core system" description.How it was tested
gofmt -l .is clean.go vet ./...is clean.go test -race ./...passes for all packages.internal/agent/agent_test.go:TestSustainSamples: conversion table (60s at 15s/5s/1s/45s/2m, zero interval, zero duration).TestCPUBurstIsSilent: the CPU threshold alerts fire on single-sample transient spikes without a sustain window #66 timeline[12%, 97%, 18%], plus a run one sample short of the window. No alert or recovery, and the key is not left active.TestCPUSustainedFiresOnce: fires only on the N-th sample, stays quiet while held, resolves on the first cool sample.TestMemSustained: a memory burst is silent, sustained pressure fires.TestSustainScalesWithInterval: atsample_interval = 15sit fires on sample 4, not 3.TestThresholdAlertPushed, the tuned-threshold check in the settings test, andTestSnoozeCallbackto run the full sustain window (via a smallevalSustainedhelper), so they still exercise a real fire.Sustainforced back to 1, the four new behavior tests fail. With the fix, they pass.Checklist
go test ./...passesgo vet ./...is cleangofmt-formatted