Health check: only alert when down, and stay quiet when powered off - #24
Merged
Merged
Conversation
Only DOWN posts now. A healthy server is silent, and a merely lossy one is silent too unless --alert-lossy is passed. Routine traffic in an alert channel is how people stop reading it, and then the message that mattered is missed as well. The check now asks DigitalOcean whether the droplet is meant to be running before deciding anything is wrong. Powering it down on purpose used to produce red alerts that were not faults. If it is off, it says so once, in grey, and stops. The DO token is optional and an unreachable API returns None rather than a guess, so it can never suppress a real outage alert. Legion step, three fixes to the hand-added version: owner is Wraith-security not wraith_security (the latter does not resolve and the workflow fails at startup); pinned to a full commit SHA rather than @v1, since a tag can be repointed and this job holds a webhook; and moved before checkout, because after it the checkout itself runs unmonitored and that is the step pulling remote content.
This file contains hidden or bidirectional Unicode text that may be interpreted or compiled differently than what appears below. To review, open the file in an editor that reveals hidden Unicode characters.
Learn more about bidirectional Unicode characters
Sign up for free
to join this conversation on GitHub.
Already have an account?
Sign in to comment
1 participant
Add this suggestion to a batch that can be applied as a single commit.This suggestion is invalid because no changes were made to the code.Suggestions cannot be applied while the pull request is closed.Suggestions cannot be applied while viewing a subset of changes.Only one suggestion per line can be applied in a batch.Add this suggestion to a batch that can be applied as a single commit.Applying suggestions on deleted lines is not supported.You must change the existing code in this line in order to create a valid suggestion.Outdated suggestions cannot be applied.This suggestion has been applied or marked resolved.Suggestions cannot be applied from pending reviews.Suggestions cannot be applied on multi-line comments.Suggestions cannot be applied while the pull request is queued to merge.Suggestion cannot be applied right now. Please check back later.
Follow-up to #23, addressing both things you asked for plus three fixes to the Legion step.
1. Only posts when DOWN
alert_lossychosen on a manual run)An alert channel is only useful if a message in it means something needs attention.
2. Quiet when the droplet is deliberately off
The check now asks DigitalOcean whether the box is meant to be running before deciding anything is wrong. Powering it down on purpose used to mean a stream of red alerts that were not faults. Now it says so once, in grey, and stops.
The DO token is optional, and an unreachable API returns
Nonerather than a guess. That direction is deliberate: DigitalOcean being unreachable must never suppress a real outage alert.Add
DIGITALOCEAN_ACCESS_TOKENto repo secrets to enable it. Without it, everything still works, just without power-off detection.3. Three fixes to the Legion step
wraith_securitydoes not resolve. The owner isWraith-security(hyphen, capital W). As written the workflow fails at startup.@v1is a mutable tag. Pinned to the full SHA0f79230..., matchingci.yml. A tag can be repointed by whoever controls the action, which is exactly the attack the step exists to contain, and this job holds a webhook.Tested against the live server and the real webhook